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Executive Summary 


The project 


Abracadabra (ABRA) is a 20-week online literacy programme composed of phonic fluency and 
comprehension activities based around a series of age-appropriate texts. Four 15-minute sessions per 
week are delivered by a teaching assistant (TA) to groups of three to five pupils. This report 
summarises the findings of a randomised controlled trial assessing the impact of ABRA on literacy 
outcomes for Year 1 pupils. The trial also assesses the impact of an offline, paper and pencil version 
of the same intervention (referred to here at ‘the non-ICT intervention’). There were 51 participating 
schools and 2,241 pupils at randomisation, and a total of 48 schools and 1,884 pupils were included in 
the final analysis (84% of the initial pupils at randomisation). 


The trial took place between October 2014 and May 2015. Fifty-one schools were randomly assigned 
to either receive some version of the intervention or to act as a ‘control’ school delivering business as 
usual. In the schools receiving the intervention, pupils were randomised to receive one of the following 
options: (1) ABRA, (2) the non-ICT intervention, and (3) standard literacy provision. The process 
evaluation involved observing sessions to understand a variety of factors in the intervention. These 
included an evaluation of which elements contributed to successful implementation, the perceptions 
and experiences of TAs and project leads, levels of pupil engagement, and the mechanisms behind 
the estimated impacts. This was an efficacy study, due to the involvement of the developer in the 
delivery of the programme. The study was funded by the Education Endowment Foundation and 
Nominet Trust as part of a funding round focusing on the use of digital technology to improve 
outcomes for disadvantaged children. 


Key conclusions 


. The children who received ABRA, or its offline alternative were found to make two and three 
months’ progress in literacy respectively compared to the children who received standard 
provision. This positive result would be unlikely to occur by chance. 


. For both ABRA and the offline alternative, the impact for children eligible for free school meals 
and children with below average pre-test outcomes was larger than for all pupils. 


. Successful implementation contributed to a well-designed and delivered training programme 
which emphasised fidelity and consistency, enforced by ongoing support from the project team. 


. The process evaluation found that both the ICT and non-ICT interventions may be best delivered 
in groups of similar rather than mixed ability. The process evaluation also suggested minor 
changes to the intervention to make it more culturally relevant to British pupils, and to remove 
some repetition in the non-ICT programme. 


. Future research will examine whether ABRA or the non-ICT intervention can be successfully 
delivered at scale, and will look at longer-term impacts through assessing Key Stage 1 data from 
this trial. 


= os Security rating awarded as 
re are the findings? ‘ 
moMesecw g part of the EEF peer review 


These findings have moderate to high security. The trial was a three-armed randomised controlled trial 
that featured randomisation between schools that received ABRA and the non-ICT intervention and 
schools that continued normal literacy provision. It also involved randomisation within the schools that 
received interventions, between ABRA, the non-ICT intervention, and pupils that received standard 
literacy provision. The trial was large and the pupils who received the intervention were similar to the 
pupils in the comparison group; 16% of the pupils were not included in the analysis because they did 
not complete all the tests at the end of the trial. 
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What are the findings? 


Both the ICT and non-ICT treatments were found to have positive results for literacy that were unlikely 
to have occurred by chance, although the size of the non-ICT effect is considerably higher. The impact 
was higher for children eligible for free school meals (FSM) for both ABRA and the non-ICT 
intervention, with both groups making the equivalent of five months’ progress. Pupils with below 
median pre-test outcomes seemed to benefit from ABRA, whereas the non-ICT intervention seemed 
to benefit both below- and above-median pupils. Pupils that received normal literacy provision in the 
schools where the interventions took place did better than students in schools that only received 
normal literacy provision. This is consistent with the existence of spillover (or peer) effects. This 
preliminary evidence will be further investigated in future research when further data on teaching 
assistant surveys and log data becomes available. 


The process evaluation indicated that the implementation of both interventions was successful and 
benefitted from a well-designed and delivered training programme which emphasised fidelity and 
consistency facilitated by ongoing support from the project team. 


These positive findings for the ICT programme are in line with previous studies about ABRA which 
found that ABRA leads to improvements in literacy. This evaluation adds to this body of evidence 
through (1) looking at a larger number of students than previous studies, (2) comparing ABRA with a 
non-ICT treatment, and (3) looking at the results of students that did not receive ABRA but went to 
school with students that did allowing the study to see if there are benefits to students that do not 
receive the intervention directly. The comparison between ABRA and the non-ICT intervention, which 
used the same materials in a non-digital format, is particularly useful as the positive non-ICT results 
seem to support the idea that it is the literacy programme itself making the difference rather than the 
digital delivery format. There are several studies that find no evidence of ICT programs having a 
positive impact on pupil outcomes using strong research designs at a relatively large scale. In this 
broader literature, the findings of this study are more unusual. 


How much does it cost? 


The average cost per pupil per year over three years is £8.52 for the ICT intervention and £8.49 for 
the non-ICT intervention. This cost includes training the teaching assistants, cover during training, and 
travel costs. All of the costs are frontloaded into the first year of the programmes (which costs £25.56 
and £25.47 per pupil respectively). The programme is free to deliver in the following years. 


Table 1: Executive Summary Table 


Effect Size Estimated Security rating 
(95% months’ 


confidence) progress 


0.138 
ied (0.004, 0.273) 


Intervention Number of 
fei ce}t] 0) ESTeq solo) f= 


0.368 
or FM 0.08 0.46 

0.231 
Non-ICT (0.102, 0.360) 

0.396 
Nort ESM 0.185, 0596) 
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1. Introduction 


Intervention 


This report summarises the findings of a randomised controlled trial assessing the impact of two 
literacy-related interventions: we analyse the effects on literacy of two methods of small-group tuition 
for Year 1 pupils in English schools. One of these methods used an ICT programme (Abracadabra) 
which has some support from small RCT efficacy studies in other countries (See background evidence 
section below). The other method is identical (using materials that replicate the ICT intervention) but 
delivered without using the computer programme (non-ICT intervention). Both methods were peer- 
reviewed in advance of implementation. 


The content of the ICT intervention was entirely encapsulated within the Abracadabra (ABRA) web- 
based technology platform, a modular ‘game’-based balanced literacy intervention that is fixed in 
content (new activities cannot and will not be added). The games are related—and are linked in terms 
of content—to a series of electronic texts (mainly ‘stories’, some non-fiction) suitable to beginner 
readers. Phonic fluency and comprehension activities are connected to, and draw most content from, 
a series of electronic texts. There are extension activities for some of the tasks within ABRA and these 
can be found in the Teacher area of the site. There are also activities for writing, but the 
implementation team chose not to include these in the 20-week programme. However, additional 
extension activities were provided for the comprehension tasks in the form of a printed sheet of ideas 
that the implementation team went through with the TAs on one of the visits. A summary of the 20- 
week ICT intervention can be found in Appendix A. 


The non-ICT intervention is also defined by a 20-week programme of lesson plans based on pencil 
and paper activities with additional resources such as magnetic letters and cards. It was especially 
developed for this study as a paper and pencil version of the ICT intervention: each of the ABRA 
activities was matched by a paper/pencil version (or using magnetic letters, for example). It also used 
all the same stories, vocabulary items, questions, words, and letter sounds in all the activities. It was 
therefore almost identical in content to the ICT version and only differed in terms of delivery medium. 
Appendix B summarises the non-ICT intervention together with the 20-week programme. The rationale 
for including the non-ICT intervention in parallel to the ICT intervention was to discern how important 
the mode of delivery was to its success, as opposed to the pedagogy underlying both approaches. 


Robert E. Slavin, director of the Center for Research and Reform in Education at Johns Hopkins 
University, conducted an independent review of the two interventions on behalf of the Education 
Endowment Foundation which concluded that they appeared to be comparable in content. Following 
his suggestions, training for both the ICT and non-ICT conditions were set to be equal in term of time. 
Training was also closely matched in terms of content but tailored for each specific mode of treatment 
delivery. Each TA was trained for 1.5 days prior to involvement, in groups of 12-13. Moreover, on 
average each TA received approximately 0.6 days of further support from the project team (a mix of 
in-person, phone, and email support). 


Training occurred after baseline testing and randomisation and prior to the start of the intervention. For 
each school, a TA was assigned randomly to the ICT or non-ICT condition before the training event.* 
A small number of big schools had two TAs per condition. Both the ICT and non-ICT TAs received 
detailed training packs after the training sessions with a description of the activities and why they were 
useful. The package included the 20-week plan (available on request) outlining the activities to be 
performed four days per week during the 15-minute sessions. Moreover, the implementation team at 
Coventry provided just-in-time support to both groups of TAs on request, and they visited the TAs 


*The school, pupil and TA randomisations were performed using the statistical programme STATA. 
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during the first weeks of treatment to observe how the intervention was being delivered and to provide 
support for the TAs. During training TAs received a list of pupils assigned to them. Prior to the start of 
the intervention TAs had some flexibility in arranging the small groups of pupils (around three to four 
pupils per group). The purpose of doing so was to give them the flexibility to divide pupils into 
appropriate groups, as they normally would do for any other activity. In practice, TAs grouped pupils 
into groups of three to four according to whether they were likely to be able to work well together. This 
was guided by ability, behaviour, SEN status, and personality. Finally, the process evaluation found no 
issues of concern over implementation or fidelity in delivery. The intervention was found to be well 
understood by TAs and implemented as intended. This included aspects such as timing, use of 
materials and organisation, and practical matters. As far as was practically possible, the two 
interventions were delivered separately without contamination in the active schools. 


The design included two control groups. One group consisted of pupils in purely control schools, and 
the second group included pupils in the control group of treatment schools. This design allows us to 
investigate the existence of spillover effects from treatment to control pupils in those schools randomly 
assigned to treatment. Spillover effects may exist, for example, because teaching assistants interact 
with pupils at other times of the school day (they were not employed only to implement the 
intervention). It is possible that the training might help them in their role generally, which would then 
impact on the control group. 


Background evidence 


Many children leave primary school without the expected level of literacy. This matters because it has 
an impact on subsequent educational performance and is linked, for example, to poor labour market 
outcomes (Hansen and Vignoles, 2005). As discussed by Slavin et al. (2011), while much is known 
about the characteristics of struggling readers, much less is known about practical, replicable 
programmes capable of helping educators prevent or remediate early reading deficits. Of those 
programmes covered in their systematic review, the vast majority pertain to the U.S. 


As explained in the previous section, we intend to analyse the effects of two methods of small-group 
tuition for Year 1 pupils in English schools. The ICT programme (based on the Abracadabra Web- 
Based Literacy Program) has had some support from small RCT efficacy studies in other countries, ° 
but the non-ICT intervention was especially developed for this study and designed to be identical in 
content to the computer-based programme without using ICT. Both interventions are balanced and 
structured reading programmes that contain a systematic phonics aspect consistent with current UK 
government recommendations. 


The ABRA programme is a freely available computer-based programme, widely used in Canada 
(Abrami et al., 2010). The programme is not used extensively in Britain, to our knowledge. However, 
the identity of the computer programme was concealed from all schools to prevent it being used in 
control schools. 


The ABRA programme has had some support from smaller efficacy RCTs (see, for instance, 
Comaskey, Savage and Abrami, 2009; Savage et al., 2009; and Wolgemuth et a/., 2011) and a bigger 
effectiveness trial (Savage et al., 2013). Savage et al. (2009) randomly allocated 174 pupils into three 


‘ Currently 11% do not reach level 4 in reading by age 11—the expected standard according to the National 
Curriculum. 

3 There was no formal piloting in the UK. However, in Coventry University a student tested out ABRA in the 
context of a small-scale intervention involving 25 Year 1 children from one school, with the control group doing a 
maths intervention (although total literacy teaching time was equated across the ABRA and control group). The 
interventions lasted for six weeks (12 sessions) and the ABRA group significantly increased their single word 
reading scores (using the British Ability Scales II single word reading sub-test) both at post-test, and after a four- 
week delay (delayed post-test). This work is not published and was produced in 2012. 
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groups: a synthetic phonics intervention group, an analytic phonics intervention group, and a 
classroom control group. The intervention groups both used the computer-based ABRA programme. 
The authors state that effect size analyses confirmed that both interventions had a significant impact 
on literacy at post-tests. The later study (Savage et al., 2013), describing a classroom-level RCT with 
just over 1,000 pupils where the intervention was delivered by teachers, also found improvements in 
literacy for treatment pupils at post-test.* The effectiveness trial in this report differs from this later 
study in several respects: (1) the size of the trial in terms of pupils is doubled, (2) this intervention 
compares ICT and non-ICT delivery of a broadly identical programme—this allows the evaluation of 
different delivery methods with respect to improving results for early literacy, and (3) we have a clean 
control group—pupils in schools that do not receive and do not know about the existence of the web- 
based programme while the intervention is in place. If we have reasons to believe that there might be 
spillover effects from treatment to control individuals within the same school, then having a control 
group in schools that are not treated gives additional information not found in the previous 
effectiveness trial (which only had a control group within the same school as the treatment group). 


Evaluation objectives 


The evaluation is designed to answer the following questions, as described in the protocol: 


1. Do the interventions have an impact on the treatment groups relative to the control group on 
outcomes within one academic year? 

2. How do the interventions compare to each other in terms of effectiveness? 

3. Is there evidence of heterogeneity in the effects of the interventions on subgroups relating to 
free school meal status, gender, whether pupils speak English as a first language, and above- 
median attainment on pre-test?° 

4. Is there any evidence of spillover effects on the control group? 

5. Do any effects persist one year later? 


Except question five for which data is not yet available (Key Stage 1 data for the cohort under study 
will be added as an appendix in Spring 2017) all the other questions are addressed in the Results 
section. 

The purpose of the process evaluation was to assess the fidelity of the intervention and to help explain 
impact of small-group tuition as measured by the RCT. We aimed to achieve this largely through an 
assessment of the practices and experiences of teaching assistants in participating schools and 
observation of the two interventions—ICT and non-ICT—focusing on teaching and learning processes 
and particularly pupil response and engagement. We carried out our own process evaluation using 
qualitative research methods and also used quantitative data collected by the project team, including 
their assessments of fidelity and quality of delivery. 


Project team 


The project team who developed and delivered the intervention are based at Coventry University and 
led by Professor Clare Wood and Dr Janet Vousden. The other team members are Rob Savage, 
Helen Johnson, Sabrina Ammi, Claire Pillinger and Sam Waldron. The ICT 20-week programme was 
developed by a team led by Dr Robert Savage, Associate Professor at McGill University. 


7 Savage et al. (2013) does not contain all the information to appropriately calculate effect sizes. However, if you 
calculate the approximate effect size for the outcome CTOPP blending words using the SD of the pre-test for the 
control group (which is similar to the SD of the pre-test for the treatment group), you get an effect size of 
1.08/3.04 = 0.36, which is in line with previous work. It is reasonable to report, in general, a ballpark effect size of 
0.3-0.4. However, the effect sizes also vary according to each outcome, for example, larger effect sizes are often 
observed for measures involving phonological processing (Such as letter-sound knowledge, blending, 
segmenting, and so on, as opposed to, say, single word reading or comprehension). 

7 Heterogeneity according to the first language spoken by the pupil cannot be assessed due to a lack of data. See 
the results section for more information on this issue. 
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The evaluation team is formed by Professor Sandra McNally and Dr Jenifer Ruiz-Valenzuela at the 
Centre for Economic Performance, London School of Economics, and Dr Heather Rolfe at the 
National Institute of Economic and Social Research (NIESR) who led the work on the process 
evaluation. 


Ethical review 


The Coventry University Ethics Committee reviewed and approved this study. Headteachers and 
relevant teachers were given a written outline of the project and asked to sign consent forms for their 
schools to be part of the project. A participant information sheet was sent out to all parents giving 
information about the project and the evaluation. Parents were asked to complete a reply slip saying 
whether or not they consented for their child to be part of the project. They were told that a non- 
response would be taken to mean no objection (therefore opt-out consent). A copy of the letters sent 
to both headteachers and parents can be found in Appendix C. 


Trial registration 


The trial was registered at: 
http:/www.controlled-trials.comlisrctn/submission/ 


The registration number is ISRCTN18254678, and the link to the ISRCTN registry website for this trial 
is: 


http:/Iwww.isrctn.com/ISRCTN18254678?q=&filters=conditionCategory:Not%20Applicable&sort 
=&offset=130&totalResults=707&page=13&pageSize=10&searchType=basic-search 
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2. Methods 


Trial design 


The results in this report are based on a randomised controlled trial (RCT) with randomisation taking 
place at the school (first stage) and at the pupil (second stage) level. Teacher assistants were also 
randomised into different treatments. In the first stage, recruited schools were paired according to the 
following characteristics: size of the Year 1 cohort, Key Stage 1 average point score for the 2013 
cohort, and percentage of FSM pupils. Additionally, infant schools were paired together. Each school 
within each pair was randomised to cohort 1 (those receiving the treatment(s) in Year 1) or cohort 2 
(those receiving the treatment(s) in Year ra In the second stage, within treatment schools, children in 
Year 1 were randomly allocated into three groups: 


(A) support from teaching assistants using the computer programme (ICT intervention); 
(B) support from teaching assistants using other methods (non-ICT intervention); and 
(C) control group within treatment schools. 


The design, therefore, included two control groups: one group consisted of pupils in purely control 
schools, and the second group included pupils in group (C) described above. This design allowed us 
to investigate the existence of spillover effects from treatment to control pupils in those schools 
randomly assigned to treatment. Finally, teaching assistants in treatment schools were randomised to 
deliver the ICT or the non-ICT intervention. 


This design was chosen in order to overcome the problem of selection bias.’ As Duflo et al. (2007) 
point out, selection bias can be entirely removed when individuals or groups of individuals are 
randomly assigned to the treatment and comparison groups. 


There were no major changes to the original trial design, but some minor changes took place post 
randomisation and before the treatment started. These are detailed in Section 3 (under the subsection 
on Participants). 


Outcome measures 


The primary outcome was measured (pre- and post-treatment) by the Progress in Reading 
Assessment (PIRA) test. This is a test that evaluates the general reading ability of pupils. ® In 
particular, it assesses reading ability in the following areas: phonics, literal comprehension, and 
reading for meaning. It is a favoured test for the Education Endowment Foundation due to having a 
marking key that the organisation feels is unambiguous and not sensitive to interpretation. This is a 
well-known test, often used in EEF evaluations. The advantages mentioned above and the fact that 
overall reading attainment is the cognitive outcome of most importance to interventions of this type 
were the reasons why the PIRA reading test was chosen as the primary outcome measure. The 
secondary outcomes assess more specific components of reading. 


® Note that the equivalent cohort in control schools will not get the treatment at any stage. Cohort 2 is a year 
younger. This enables us to consider the effects of the treatment on longer-term outcomes (as indicated by Key 
Stage 1 teacher assessments). 

’ Selection bias arises when individuals or groups are selected for treatment based on characteristics that may 
also affect their outcomes and makes it difficult to disentangle the impact of the treatment from the factors that 
drove selection (Duflo et al., 2007). 

8 More information on the PIRA test can be found at https:/www.hoddereducation.co.uk/pira 
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9 
The secondary outcomes that have been measured are as follows: 


e DTWRP (the Diagnostic Test of Word Reading Processes). This assesses the reading of 
regular words, exception words, and non-words, enabling the nature of a child’s reading 
difficulty to be easily identified. 

e BPVS (the British Picture Vocabulary Scales, Third edition). This is a one-to-one test that 
assesses a child’s receptive vocabulary. For each question, the teacher says a word and the 
pupil responds by selecting a picture from four options that best illustrates the word’s 
meaning. 

e PSS (the Phoneme Segmentation Subtest) and LK (the letter sound subtest)—subtests from 
the Primary Inventory of Phonological Awareness (PIPA) test. PIPA assesses the nature and 
extent of a child's phonological awareness development. 

e LEST (the Letter Sound Test). This tests a person’s ability to sound out single letters and 
letter combinations. 


All of these secondary outcomes were included as secondary measures in the protocol except the 
LEST test which replaced the PIPA test originally specified in the evaluation protocol. The 
implementation team at Coventry examined the tests and realised that the letter sound subtest (LK) 
on the PIPA test was very basic (consisting of mainly single letters and no vowel digraphs such as 
‘ea’ or ‘ou’). Given that in the UK most children receive phonics instruction from the Reception Year, 
they learn the single sounds fast and are often near or at the ceiling by the end of Year 1. Moreover, 
children often score very high marks at the beginning of Year 1. The single letter sounds are also 
much less predictive of word decoding. The LEST is much more comprehensive and includes the 
most frequently used English digraphs (‘th’, ‘sh’, ‘ea’, ‘ou’, ‘ow’, ‘gh’ and so on). Therefore, the 
implementation team at Coventry decided to include the additional LEST test. 


These tests took place at the beginning (September 2014, all before randomisation) and end (July 
2015) of the academic year 2014/2015. 


We will also analyse secondary outcomes such as Key Stage 1 outcomes on reading and writing one 
academic year later (as data becomes available in September 2016). 


Assessments were administered by a team of Research Assistants (RAs) employed by Coventry 
University who did not know to what condition the children had been allocated. Furthermore, the RAs 
were blind to the nature of the study, in other words, they were not given any details about the project 
other than it was a reading project. Assessments were administered individually or in small groups 
with spot checks by the project team to ensure correct administration. The baseline PIRA assessment 
was scored by Hodder Education. All other tests were scored (and entered) by a group of RAs hired 
specifically for this purpose (not those who carried out the assessments), with no knowledge of how 
schools or pupils had been allocated to the treatment and control groups, and no knowledge of the 
nature of the project other than it was a reading project. Data-checking was carried out by the RAs 
who marked the assessments, and again by the project team. In particular, the implementation team 
conducted further checks on the data like range checks, age, date of birth, and date of test checks, 


° More information on these secondary outcomes can be found here: 

DTWRP: http://www.gl-assessment.co.uk/products/diagnostic-test-word-reading-processes 

BPVS: http://www.gl-assessment.co.uk/products/british-picture-vocabulary-scale-third-edition 

PIPA: 
http://www.pearsonclinical.co.uk/Psychology/ChildCognitionNeuropsychologyandLanguage/ChildLangu 
age/PreschoolandPrimaryinventoryofPhonologicalAwareness(PIPA)/PreschoolandPrimaryinventoryofPh 
onologicalAwareness(PIPA).aspx 

LEST: http://www.motif.org.au/homelitest/7 
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checking zero scores against missing data, and so on. This was done in order to catch any mistakes 
in data entry. 


Participant selection 


The implementation team at Coventry University first selected all schools with primary-aged children 
in its geographical area—the West Midlands, where the project team is based. There was an attempt 
to recruit 60 schools in Coventry, Warwickshire, Solihull, Leicestershire, Northamptonshire, Walsall, 
Sandwell, Worcestershire, Birmingham, and the West Midlands. These are more areas than the ones 
originally stated in the protocol (Birmingham, Coventry, Warwickshire, Worcester and Solihull) in order 
to widen the net to recruit more schools. There were a total of 1,742 schools. Coventry University 
excluded schools that did not contain a Year 1 group, leaving them with 1,682 eligible schools all of 
which were invited to take part. In particular, an effort was made to encourage schools with 
disadvantaged intakes to participate during the recruitment stage. Coventry University ran a 
recruitment event where the project was explained to leaders from interested schools. Schools 
needed to sign a Memorandum of Understanding in order to participate. They were required to send 
an opt-out consent form to all parents ahead of testing in order to make a request for administrative 
data (the National Pupil Database) to the Department for Education. The consent was obtained before 
randomisation. The participation information sheet sent to parents is found in Appendix C. All of the 
approached schools that agreed to participate were included in the study. 


Sample size 


The calculations to decide on the sample size included in the protocol were performed using the 
Optimal Design (OD) Software (Spybrook et al., 2011).*° Our design paired schools with the aim of 
grouping them into clusters of similar schools. As Spybrook et al. (2011) point out, the aim of pairing 
is to reduce the heterogeneity within pairs (also called blocks or sites). This increases the precision of 
the treatment effect estimate and hence increases the power of the test. In the protocol (first row of 
Table 3) we had assumed that 60 schools would be recruited and paired according to 
characteristics.‘’ Each school within each pair would be randomised to cohort 1 (to receive the 
treatment in Year 1) or cohort 2 (to receive the treatment in Year 2). On average, schools were 
assumed to have 60 pupils in their Year 1 cohort. The predictive power of baseline characteristics 
was set at 0.7, the intraclass correlation at 0.2, the power of the test at 0.80, and the statistical 
significance of the effect at 0.05. Further assuming that the proportion of the explained variance by 
the blocking variable is 0.10 and the effect size variability is 0.01, this sample size was powered to 
detect an effect of at least 0.2 standard deviations. We did not suggest increasing the size of the 
sample to take account of potential attrition as 60 schools was already an ambitious number to recruit 
for the experiment. See Table 3 in the next section for more information related to the minimum 
detectable effect size at randomisation and at the analysis stage. 


As stated in the protocol, these are conservative estimates as they are based on a multisite cluster 
randomisation at the school level (where the sites are the pairs of schools). In practice, we also 
randomised pupils within schools. This was not incorporated in the calculations because the 
programme used (OD) does not have this option, but this further level of randomisation at the student 
level should enable us to detect smaller size effects. This is because compared to the multisite cluster 
randomisation, in our case we had a sequential randomisation process that randomised first schools 
and then students in treatment schools. The variance of the treatment effect in this case should be 


The programme is free and available to download here: 
https://sites.google.com/site/optimaldesignsoftware/home 

*? Recruited schools were paired according to the following characteristics, with the following order of importance: 
size of the Year 1 cohort, Key Stage 1 average point score for the 2013 cohort, and percent of pupils that 
received free school meals at the time. Additionally, infant schools were paired together. 
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potentially smaller. Therefore, this should enable us to estimate more precise estimates and thus 
detect effect sizes smaller than those predicted with the OD programme for the multisite cluster 
randomised trial option. 


Randomisation 


Paired randomisation was performed by the evaluation team at the Centre for Economic Performance 
(LSE) using the statistical programme STATA, whereas recruitment was carried out by the team at 
Coventry University, therefore ensuring independent allocation. 


The list of variables used to pair schools included the size of the Year 1 cohort as supplied by the 
implementation team at Coventry University, Key Stage 1 average points score for the 2013 cohort, 
and a measure of the percentage of FSM pupils provided by the implementation team.” Additionally, 
infant schools were paired together 


The random allocation sequence to assign schools to treatment or control groups was generated: 


1) by first drawing pairs of schools (we drew numbers from a uniformly distributed random 
variable) and then 

2) from each pair we assigned the school with the highest drawn number to the treatment group, 
and the school with the lowest drawn number to the control group. 


The random allocation of pupils to the three different conditions in treatment schools was achieved by: 


1) first, in each school, randomly allocating a number to each pupil in the school drawn from a 
uniformly distributed random variable, and then 

2) in each school we assigned the first third of pupils with the lowest random numbers to the 
control condition, the second third of pupils with the lowest random numbers to the non-ICT 
condition, and the remaining third of pupils with the highest drawn numbers to the ICT 
condition. 


Finally, the random allocation sequence to assign TAs in treatment schools to the ICT or non-ICT 
conditions was performed in a way analogous to the school randomisation. Codes are available upon 
request from the authors. A small number of deviations from this randomisation procedure are 
explained in the Participants subsection of the Impact Evaluation Section (see below). 


Analysis 


Our analysis follows the protocol and estimates Ordinary Least Squares (OLS) regressions for each 
outcome variable where the outcome is regressed against dummy variables for whether the pupils are 
in the ICT treatment or the non-ICT treatment. In this first analysis we control for the pairing dummies, 
as well as for prior attainment (as given by the baseline test for each outcome), gender, FSM 
eligibility, and a dummy variable indicating whether the pupil had a good level of development in the 
Foundation Stage Profile (FSP GLD).** Standard errors in the regression analysis are always 


2 The FSM measure provided by Coventry for the pairing was downloaded from the Edubase dataset, 
http://www.education.gov.uk/edubase. This FSM measure is defined as the percentage of pupils receiving 
FSM in that school year. This is calculated as the number of pupils known to be eligible for and claiming free 
school meals expressed as a percentage of all pupils. Includes pupils who have full time attendance and are 
aged 15 or under, or who have part time attendance and are aged between five and one. 

*8 ESP GLD is a dummy variable that indicates whether the pupil has achieved a good level of 
development in the Foundation Stage Profile. This is the case if the pupil achieved a level of 2 or 3 in 
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clustered at the school level. This takes into account the fact that students are clustered in schools, 
and therefore outcomes might be correlated. 


This specification is close to that stated in the protocol. Specifically, the protocol stated that we would 
control for the following baseline characteristics: baseline test score, FSM eligibility, gender, ethnicity, 
and whether English is spoken as a first language. Unfortunately, the conditions for the supply of 
ethnicity and language data at the pupil level were not fulfilled and so could not be provided by the 
Department for Education.** The decision to include the FSP GLD variable is done in order to gain 
further precision. 


All the outcome variables and baseline tests have been standardised to have mean O and standard 
deviation (SD) 1 using the mean and SD of the outcomes (we have used the mean and SD for the full 
sample for each of the outcomes, both at post-test and at baseline respectively). This allows us to 
interpret the coefficients of the explanatory variables in terms of standard deviations of the outcome 
variable. All regressions cluster standard error at the school level. 


Our analysis has been on an intention-to-treat basis—as explained in the Participants subsection of 
the Impact Evaluation, five schools dropped out after randomisation—however, for four out of these 
five schools we were able to obtain post-test data and they are included in our analysis. We check 
whether baseline characteristics are balanced for the final sample to discern whether the small 
amount of attrition (losing one school) creates imbalance in the different intervention and control 
groups. We also check whether those pupils with missing data at follow-up (post-treatment stage) had 
different baseline characteristics than the pupils for which we have data at post-test. Since we do not 
find significant differences (see the Analysis section) we do not impute missing data and work with the 
available final sample. 


As stated in the protocol, we explore whether there are heterogeneous effects of treatment on the 
main outcome variable according to FSM status, gender, and above-median attainment on pre-test. 
For example, for FSM, we do this by interacting each treatment variable with a dummy variable that 
indicates whether or not the pupil is eligible to receive FSM.*° 


Finally, the protocol specified that in order to address the possibility of spillover effects from treatment 
pupils to control pupils in treatment schools, we would run a first regression where the control group 
would be formed by pupils in the same schools, and a second regression where the control group 
would be formed by pupils in control schools only. We could then check whether the treatment effects 
were larger in the second regression than in the first one. If so, this is likely to be indicative of spillover 
effects. However, running regression one implies working only with treatment schools (the 24 
clusters). The reduction in the number of clusters is rather problematic given that we need to cluster 
standard errors at the school level. Although we did this, we first run a regression where an additional 
dummy variable is added to the full sample for pupils that are in the control group within treatment 
schools (CT variable). We compare the coefficient on CT to those of the ICT and non-ICT treatment 
dummies. 


each of COM (Communication), PHY (Physical development), PSE (Personal, Social and Emotional 
Development), LIT (Language and Literacy) and MAT (Mathematical development) results. 

™ This is because the consent forms sent to parents ahead of testing were opt-out consent forms. In order to be 
able to receive sensitive NPD data, parents should have given explicit consent (opt-in consent forms). 

*° This variable comes from the National Pupil Database, in particular from the pupil Census. A pupil is recorded 
as eligible for Free School Meals on Census day. From 2012/13, this variable set to ‘true’ (‘1’) if a pupil has an 
FSM period with a start date and end date blank or end date on or after census date which means they are FSM 
eligible on Census day. 
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Implementation and process evaluation 


The principal aims of the process evaluation of ABRA were to understand the perceptions of the 
programme of teaching assistants and project leads and to identify factors affecting its successful 
implementation through qualitative research. The evaluation aimed to identify features that appeared 
to contribute to successful implementation, including practicalities and the influence of factors such as 
staff confidence in the approach. Qualitative research within the process evaluation also aimed at 
bringing greater clarity to the quantitative research findings and understanding the reasons for, and 
mechanisms behind, the estimated impacts. 


The process evaluation aimed at understanding participants’ experiences of the programme at the 
training stage and during implementation and application of ABRA within schools. It was designed to 
include observations of ABRA sessions to gain an insight into pupil engagement with the ICT and 
non-ICT interventions: the original plan was to carry out visits in two stages—initial and follow-up 
visits. However, given the demands on TAs it was agreed with the project team that email follow-up 
would be preferable. This was particularly necessary in view of the scale of process evaluation work 
carried out by the project team, which included surveys and the collection of recording and tracking 
data. 


The process evaluation consisted of the following components: 


e Evaluation of teacher preparation and training through: 
o attendance at the introductory conference at Coventry University in May 2014; 
Oo observation of teaching assistant training in both ICT and non-ICT interventions at 
Coventry University in October 2014; 
0 analysis of project training materials issued to TAs; and 
oO analysis of training evaluation data collected by the implementation team at Coventry 
University. 


e Evaluation of implementation, fidelity, and impact through: 
0 visits to six intervention schools in January 2015 to carry out 
= interviews with 16 TAs, 
= observe 13 ICT-based sessions and five non-ICT sessions, and to 
= analyse the project resources in the context of their use; 
o follow-up surveys of the six intervention schools in June 2015 resulting in responses from 
three; 
0 analysis of survey on TA’s background and experience carried out by Coventry University 
and completed by 46 of 49 TAs; 
0 analysis of data on TAs’ recording and tracking carried out by Coventry University; 
0 analysis of data collected on TAs’ experiences of the intervention carried out by Coventry 
University, completed by 37 of 49 TAs; and 
0 analysis of data collected by Coventry University on the practices of treatment and control 
schools in relation to how they teach literacy completed by 13 of 21 active schools and 14 
of 24 in the control group. 


ABRA was implemented in the Midlands with many schools situated in Birmingham, Coventry and 
Leicester. NIESR selected six schools to include a range by size, location and performance as 
measured by Ofsted. Information about these schools and the size of the Year 1 ABRA cohort is 
presented in Appendix D. The visits and follow-up survey enabled us to gain a more in-depth 
understanding of the implementation of ABRA while our own assessment of the project team data— 
including tracking, evaluation of training, and usual practice in relation to teaching literacy—gave us a 
broader understanding of teachers’ experiences across all schools involved in the project. 
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Data, collected through the range of approaches described above, was analysed using thematic 
analysis within a framework approach.*° This enables the analysis of qualitative data in a written form 
and is therefore appropriate for the analysis of transcripts of interviews with teaching assistants and 
project leads, as well as research notes taken during observation of training. Qualitative responses to 
training evaluation surveys were also analysed in this way. The method entails coding the data into 
themes and issues. In this case, codes were a mixture of predetermined ones, developed during the 
design of the process evaluation and taking account of the aims of the intervention, as well as those 
that emerged from the text of transcripts and observations. Codes identified different types of 
information, for example, more tangible ones—such as previous use of phonics programmes, 
experiences of the training, and of putting the approach into practice—as well as others such as 
values and feelings. Throughout the analysis process we looked for similarities and differences in the 
data. The framework approach allows for tracts of text to be classified under more than one code, and 
codes were, in some cases, amalgamated to form wider groups, particularly where substantial issues 
were concerned. 


The codes and groups developed in the analysis of data formed the analytical framework and were 
used to structure the findings into a preliminary report. The framework was formed from the data from 
case-study visits which was then used to structure findings from the follow-up survey, project team 
data, and training evaluation. In the report, where relevant, we identify the source of a finding. 


In the final stage of analysis we re-structured our document to follow the format required by the EEF. 
This involved structuring the findings using the main EEF process evaluation criteria: implementation, 
outcomes, and fidelity. 


e Ritchie J. and Lewis J. (2003) Qualitative research practice: a guide for social science students and researchers, London: 
Sage. 
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Timeline 


Table 2: Timeline 


Date Activity 


January-April 2014 Coventry team invites schools ahead of a recruitment event that will take 
place the week after Easter half-term. 


Coventry team develops treatment curricula for both treatments and 
sends to CEP for review. CEP develops an assessment form for these 
curricula to send back to Coventry. 


March-April 2014 Abate organises an anonymous process of peer review of the two 


Autumn 2014 (and 
beginning of 2015 to 
correct mistakes in 
pupil identifiers and 
dates of birth) 


Coventry organises baseline tests in all participating schools and 
CEP randomises schools into cohort 1 (treatment in Year 1) and cohort 
2 (control). CEP uses year group lists to randomise pupils in treatment 
schools to groups A, B and C. Coventry sends CEP the CVs of all 
October 2014 teaching assistants taking part in the experiment (November 2014). 
CEP randomises teaching assistants to groups A and B within treatment 
schools. Coventry organises training of teaching assistants. Attended by 
NIESR and a team member of CEP. 


October 2014-May Coveniry takes responsibility for implementation of treatment in groups A 
2015 and B in treatment schools. 


NIESR conducts first round of process evaluation visits to six selected 
schools. 


June-July 2015 Coventry organises second round of testing in all participating schools. 
July 2015 ee collects follow-up data from six selected schools through a 


January-March 2014 


CEP applies for the National Pupil Database for relevant cohorts of 
pupils in all participating schools. 


January 2015 


September 2015-July Coventry organises implementation of programme in schools 
2016 participating in cohort 2. 


CEP works on the EEF evaluation report (to be extended with an 
addendum report when KS1 outcomes become available in 2016). The 
final set of data was received by mid-December 2015. 
From July 2015 CEP and Coventry to work on quantitative evaluation with a view to 
onwards publication in academic outlets. 
NIESR analyses data from interviews and observations, from training 


evaluation and from teaching assistant logs for the process evaluation 
section of the EEF report. 


Costs 


The cost information has been collected by the implementation team at Coventry (that is, data has 
been collected from the developer). The interventions cost the schools nothing in the trial. In the Costs 
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section (after the Results section of this report), we provide the costs associated with implementing 
each intervention had the EEF not funded it. This gives detailed information on costs per pupil, and 
how the current costs of this intervention could be further reduced. 
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3. Impact evaluation 


Participants 


As described in the Participant Selection subsection, the implementation team at Coventry University 
first selected all schools with primary-aged children in the geographical areas near to them. There 
were a total of 1,742 schools in the selected geographical areas. Coventry University excluded 
schools that did not contain a Year 1 group, leaving them with 1,682 eligible schools. Of the schools 
that attended the recruitment session, 58 signed contracts to participate in the intervention. Six 
schools dropped out prior to baseline testing (due to changes in circumstances) and one school 
dropped out after baseline testing as this school found the testing too disruptive. 


A total of 51 schools were part of the randomisation (randomisation took place after baseline testing 
had concluded). One of the schools randomised is a private school. In the paired randomisation, this 
private school was paired to a blank (non-existent) school in order to decide whether to assign that 
private school to treatment or control. (This school is not included in the analysis.) 


Among the 26 treatment schools, the initial number of pupils to be randomised into the three 
conditions was 1,093. Randomisation resulted in the three following groups: ICT treatment (373 
pupils), non-ICT treatment (365 pupils), and control pupils in treatment schools (355 pupils). However, 
post randomisation and before the treatment started, some changes were requested by the 
implementation team at Coventry in order to reduce drop-out of entire treatment schools. As 
evaluators we were aware of the problems of re-randomising for some particular schools however we 
granted these changes so that power would not be further compromised. These few changes 
concerned the following: 


a. Two schools had classes in Year 1 in different school locations. We were asked to re- 
randomise pupils to be able to form groups that would come from the same locations 
(otherwise, it would have been impossible for them to form the treatment groups). 

b. We were asked to reduce the number of pupils in the treatment conditions or the school would 
drop out. This happened in two cases (two schools). In one case, ten pupils had been 
assigned to each treatment and we had to reduce the number to four in each treatment 
instead. In the second case, there were 14 pupils initially allocated to the ICT and non-ICT 
conditions respectively. We were asked to reduce the number of pupils in each condition to 
12. 

c. A small number of pupils had left the school or had withdrawn consent so they should not 
have been included in the randomisation. We randomly assigned pupils from the original 
control group to treatment only when schools pressed to have treatment groups of equal size. 
This happened for two schools (one pupil in each school was randomly chosen from the 
original control condition to join the corresponding treatment group). 


The number of pupils in the sample of treatment schools after these changes was 1,083 (we lost ten 
pupils: four because they had withdrawn consent and six school leavers). The final distribution of 
pupils in treatment schools before the start of treatment was: ICT treatment (360 pupils), non-ICT 
treatment (350 pupils), and control pupils in treatment schools (373 pupils). We do not believe that 
these changes represent a threat to validity for two main reasons. First, the number of students 
involved in the changes is small. Second, all the changes described in points a, b, and c above were 
done by allocating pupils at random to meet the requirements. 


There were initially 1,158 pupils in control schools. Together with the 1,083 pupils randomised to the 
three different conditions in treatment schools, this made an initial sample of 2,241 pupils. 
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A small number of pupils—26 pupils from treatment schools and 35 pupils from control schools—did 
not sit the baseline PIRA test (pupils sat several tests in different dates, both for baseline and post- 
intervention tests—we focus here on PIRA, the primary outcome).*’ 


Likewise, not all pupils in the randomised sample sat the post-intervention PIRA test (the follow up 
tests after the intervention had finished). From the original 1,083 (1,158) pupils in the treatment 
(control) sample, 977 (1,104) pupils sat the post-intervention PIRA test. The reasons why some pupils 
did not sit the test can be found in Figure 1. 


A total of five schools dropped out from the intervention post-randomisation, all of them in the 
treatment group. Three schools dropped out right after randomisation took place (two because they 
could not see how to integrate the intervention with their current literacy provision and worried that the 
children might get confused, and one for unknown reasons), and two dropped out during the 
intervention (one in November 2014 due to staffing, the other one in February 2015 due to a change in 
the head teacher). We advised the implementation team to collect data for all initially randomised 
schools. They were able to collect post-intervention data for four out of the five schools that dropped 
out after randomisation. This allows us to perform an intention-to-treat analysis using most of the 
original randomised schools. However, given that we used paired randomisation, we remove from the 
main analysis both the school for which we did not get any post-test data and its pair, as well as the 
independent school that was paired to a blank school. We are aware that this decision could introduce 
selection bias and reduce statistical power. However, when we include the full sample of schools (with 
the exception of the one school for which we don’t have any post-intervention test data), and exclude 
the pairing dummies as controls, the results remain virtually unchanged (results available upon 
request). 


As a result of excluding these three schools from the main analysis, and also from having to exclude 
pupils without PIRA tests both at both baseline and follow-up, we are left with a total of 1,901 pupils 
(877 in treatment schools and 1,024 in control schools), almost 85% of the randomised sample. If we 
were to include all pupils for whom we have pre- and post-intervention test data, we would work with 
1,969 observations, almost an 88% of the randomised sample. Additionally, the FSM _ indicator 
(measuring whether pupils were FSM eligible on Census day) provided by the National Pupil database 
is only available for 1,887 pupils, almost 84% of the randomised sample. Thus, attrition in this study is 
lower than 20% as 84% of the randomised pupils were available in the final analysis. 


*’ The reasons why pupils did not take the baseline PIRA test in treatment or control schools included 
absence at the time of the test (14/28), being unable to access test due to behaviour, EAL status, SEN 
status, and so on (3/4), blank test returned (might be because pupil refused to take the test or was 
absent) (6/0), children leaving school (1/0), unknown reasons (2/2), and clerical error (0/1). 


Education Endowment Foundation 20 


ABRA: Online Reading Support 


Figure 1: Number of schools and pupils at different stages of the RCT 


Approached 
(school n = 1,742) 
Excluded (school n = 60) 


because they did not have a 
Year 1 group 


Assessed for eligibility 


Signed a contract to 
(school n = 1,682) 


participate (school n = 58) 
Decline to participate before 
baseline test (n = 6) 

Decline to participate after 
baseline test (n = 1) 


Recruitment 


Randomised 
(school n=51; pupil n=2241) 


Allocated to intervention Allocated to control 
(school n = 26; pupil n = 1,083) (school n = 25; pupil n = 1,158) 
Pupils sitting baseline PIRA test (n = 1,057) Pupils sitting baseline PIRA test (n = 
Pupils NOT sitting baseline PIRA (n = 26; 1,123) 
reasons, see text) Pupils NOT sitting baseline PIRA (n = 35; 
ICT (360); reasons, see text) 
Non-ICT (350); 
Control in treated schools (373) 


Allocation 


Lost to follow- Post-test data Post-test data Lost to follow- 
up (n = 166): collected for PIRA collected (pupil n = up (n = 54): 
test (pupil n = 917): 1,104) 
Absent (10); Absent (6); 
moved schools moved schools 
(41); school (47); clerical 
withdrew (114); error (1) 
unknown (1) 


Not analysed, Analysed (school n = Analysed Not analysed 
i.e. excluded 24; pupil n = 877): (school n = 24; pupil (school n= 1; 
from the 917 n= 1,024) pupil n = 80) 
observations ICT (290); 

above (school Non-ICT (277); 

n = 2; pupil n= Control in treated 

40) schools (310) 
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With these numbers, and making some assumptions, we can calculate the minimum detectable effect 
size at different stages of the trial. Table 3 summarises these calculations, performed using the 
Optimal Design Software (Spybrook et al., 2011); ° Our design paired schools with the aim of grouping 
them into clusters of similar schools. The list of variables used to pair schools includes the size of the 
Year 1 cohort as supplied by the implementation team at Coventry University, Key Stage 1 average 
points score for the 2013 cohort, and a measure of the percentage of pupils classified as FSM-eligible 
provided by the implementation team. Additionally, infant schools were paired together. 


As Spybrook et al. (2011) point out, the aim of pairing is to reduce the heterogeneity within pairs (also 
called blocks or sites). This increases the precision of the treatment effect estimate and hence 
increases the power of the test. In the protocol (first row of Table 3) we had assumed that 60 schools 
would be recruited and paired according to characteristics, and that each school within each pair 
would be randomised to cohort 1 (treatment in Year 1) and cohort 2 (treatment in Year 2). On average, 
schools were assumed to have 60 pupils in their Year 1 cohort. The predictive power of baseline 
characteristics was set at 0.7, the intraclass correlation at 0.2, the power of the test at 0.80, and the 
statistical significance of the effect at 0.05. Further assuming that the proportion of the explained 
variance by the blocking variable is 0.10 and the effect size variability is 0.01, this sample size was 
reasonable to detect an effect of at least 0.2 standard deviations. Under the same assumptions, but 
considering the number of schools at randomisation, this design allows detecting an effect of at least 
0.22 standard deviations. Finally, with the data analysed, we can calculate some of the assumed 
parameters (see information in the table). With these new values, we can detect effects of at least 
0.27 standard deviations with a power of 0.80. 


These are conservative estimates as they are based on a multi-site cluster randomisation at the 
school level (where the sites are the pairs of schools). In practice, we have also randomised pupils 
within schools. As stated in the protocol, within the schools assigned to the treatment in Year 1, pupils 
have been randomly allocated into three groups: (A) support from teaching assistants using the 
computer programme (ICT intervention), (B) support from Teaching Assistants using other methods 
(non-ICT intervention), or (C) a ‘business as usual’ control group. This should enable us to detect 
smaller size effects, as explained in more detail in the section above. 


*® The programme is free and available to download here: 
https://sites.google.com/site/optimaldesignsoftware/home 
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Table 3: Minimum detectable effect size at different stages 


N Correlation Minimum 
[schools/pupils] pe Cee. Blocking/stratification eho ti 
ae ee test (+other : : Power Alpha_ eeffect 
(n=intervention; : or pair matching : 
n=control) covariates) & size 
post-test (V1 2) S59) 
Schools: 60 (30; 
30) School blocking (30 
Protocol | Pupils in Year 1: ; : blocks with 2 schools by | 80% 0.05 0.20 
60 on average pair) 
(by school) 
Schools: 50 (25; 
25) School blocking (25 
Random | pupils in Year 1: .20 | blocks with 2 schools by | 80% | 0.05 0.22 
isation : 
60 on average pair) 
(by school) 
Analysis | Schools: 48 (24; 
(availabl 24) School blocking (24 
e pre- & | Pupils in Year 1: 0.43 0.15 | blocks with 2 schools by | 80% 0.05 0.27 
post- 40 on average pair) 
test) (by school) 


Pupil characteristics 


Table 4 shows whether pupil characteristics were balanced at baseline for the analysis sample using 
the maximum number of observations available for each variable. Outcome variables are standardised 
to have a mean of O and a standard deviation of 1 (enabling us to interpret estimates in terms of 
standard deviation impacts). Each column shows the results of regressing each characteristic on a 
dummy variable that takes on the value of 1 if the pupil is enrolled in a treatment school. The constant, 
therefore, shows the average value of that characteristic in control schools, and the dummy variable 
coefficient picks up the difference for treatment schools. Columns 1 and 2 show that there are no 
significant differences between pupils in treatment and control schools in terms of gender and FSM 
eligibility, and column 4 shows that the pupils who did not sit the PIRA test at baseline (before 
randomisation) are not missing differentially in treatment and control schools. Column 3 shows that 
pupils in treatment schools were less likely to have achieved a good level of development (GLD) in the 
Foundation Stage Profile (FSP). Controlling for the FSP development level between treatment and 
control pupils has only a minimal effect on point estimates for treatment effects. However, it does 
improve the precision of the estimates. Thus we include this as an additional control above that 
specified in the protocol. 


Table 5 repeats the same exercise using all the outcome variables measured at baseline. In none of 
these cases do we find significant differences in the baseline tests between pupils in treatment and 
control schools. 
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Table 4: Balance at baseline (final sample)—pupil characteristics 


C2) | )) C) 


FSM Missing PIRA 
eligibility Fer CEP at baseline 
=1 if school is a treatment -0.005 -0.000 -0.087*** -0.005 
school (0.017) (0.023) (0.022) (0.008) 
Ganciant 0.519*** 0.458*** 0.455*** 0.021* 
(0.019) (0.133) (0.044) (0.011) 
Observations 1,901 1,887 1,893 2,241 


Note: Standard errors clustered at the school level in parentheses (* p < 0.10; ** p < 0.5; *** p < 0.01). FSM eligibility: pupil 
recorded as eligible for free school meals on Census day. FSP GLD: pupil has achieved a good level of development—achieved 
level of 2 or 3 in each of COM, PHY, PSE, LIT and MAT results (See main text). All regressions include pairing dummies 
(schools were paired for randomisation). 


Table 5: Balance at baseline (final sample) - Outcome variables measured pre-randomisation 


9) (4) EC) (3) (3) 
PIRA DTWRP =] =AVAS) | PS LK LEST 
=1 if school is a treatment -0.061 0.001 -0.101 0.046 0.030 -0.061 
school (0.068) (0.056) (0.066) (0.048) (0.063) (0.061) 
een -0.383*** -0.026 -0.139 -0.139*** 0.039 0.111 
(0.085) (0.176) (0.134) (0.050) (0.085) (0.251) 
Observations 1,901 1,897 1,872 1,929 1,929 1,916 


Note: Standard errors clustered at the school level in parentheses (* p < 0.10; ** p < 0.5; *** p < 0.01). All regressions include 


pairing dummies (schools were paired for randomisation). All tests (outcome variables) are standardised at the baseline level to 
have mean 0 and SD 1, using all the observations available at baseline. 


The last row in Table 5 shows the number of observations available in the final sample for each of the 
outcome variables. As documented above, there are 1,901 pupils in 48 schools (clusters) available for 
the main outcome (the PIRA test), and this represents almost 85% of the randomised sample. For the 
other secondary outcomes, the number of pupils available varies slightly with respect to the main 
outcome sample as can be seen in the last row of Table 5. The number of observations available 
represents 85% (DTWRP), 84% (BPVS), 86% (PS), 86% (LK) and 85% (LEST), respectively, of the 
randomised sample. Table 6 explores whether pupils who were unable to be followed up from 
baseline tests are different from those for whom we do have post-treatment data. Columns 1 to 4 
show that there are no statistically significant differences in terms of gender, FSM eligibility, FSP 
development level, and PIRA results at baseline. Moreover, column 5 shows that pupils that we could 
not follow up do not differ in terms of baseline PIRA results in treatment and control schools (see the 
coefficient for the interaction between the variable ‘Missing pira at post-test’ and the dummy variable 
for treatment schools). Finally, column 6 repeats the analysis in column 5, distinguishing between the 
two types of treatment (ICT and non-ICT). This shows that pupils who do not have a post-intervention 
test score are no different from other pupils, conditional on what group to which they are assigned. 
Given these findings, the tables shown in the next section use the final available sample without 
imputing missing values. 
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Table 6: Analysis of missing data at follow-up 


C2) 


i) 
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(4) 


) 


) 


FSM FSP ——~wPlRAat PIRA at PIRA at 
eligibility (ej Hp) baseline baseline baseline 
Missing PIRA at post- -0.009 -0.087 0.018 0.064 0.144 0.068 
test (0.036) (0.076) (0.047) (0.094) (0.116) (0.091) 
=1 if school is a -0.043 
treatment school (0.074) 
Missing Bees at post- 0.096 
test* (= 1 if school isa (0.189) 
treatment school) ; 
-0.053 
ICT 
is (0.082) 
-0.043 
NONICT 
one (0.074) 
ICT* (= Lif schoolisa -0.011 
treatment school) (0.151) 
NONICT* (= 1 if school 0.031 
is a treatment school) (0.139) 
Consent 0.482*** 0.482*** -0.027 0.003 -0.013 
(0.020) (0.037) (0.155) (0.167) (0.157) 
Observations 2180 2143 2149 2180 2180 2180 


Note: Standard errors clustered at the school level in parentheses (* p < 0.10; ** p < 0.5; *** p < 0.01). Missing PIRA at post-test 
is a dummy variable that takes on the value of 1 if we don't observe PIRA at post-test. FSM eligibility: pupil recorded as eligible 
for free school meals on Census day. FSP_GLD: has achieved a good level of development—achieved level of 2 or 3 in each of 
COM, PHY, PSE, LIT and MAT results. All regressions include pairing dummies (schools were paired for randomisation). PIRA 
at baseline is standardised to have mean 0 and SD 1. ICT and NON-ICT are the treatment dummies. 


Finally, in Table 7 we provide the baseline demographic, school-level, attainment, and other relevant 
characteristics of the final sample for the intervention and control schools. The upper panel of Table 7 
presents means of several school characteristics for treatment and control schools, all measured with 
data for the academic year 2013/2014 and for the final (analysis) sample. In particular, information is 
given on the type of school, whether the school is rural or urban, Ofsted overall effectiveness 
according to the last inspection until August 2014, the IDACI rank, and the Index of Multiple 
Deprivation. The bottom panel presents information on other continuous variables at the school level 
(percentage FSM, SEN, and so on). Although there are some differences between the treatment and 
control groups (such as in the distribution of schools according to the IDACI rank or Index of Multiple 
Deprivation), there is a reasonable and broadly comparable spread of schools within the categories 
considered here. 
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Table 7: Baseline comparison (school-level variables for the final sample, 2013/2014 data) 


WEVar=10) (<1 aitciaV{-laid(olaMe| celle) Control group 


School-level (categorical) n/N (missing) Percentage n/N (missing) Percentage 


Academy (converter/sponsor 
led 


3/24 (0) 12.5% 6/24 (0) 25% 


Community school 11/24 (0) 45.8% 9/24 (0) 37.5% 


Voluntary (aided/controlled) 10/24 (0) 41.7% 9/24 (0) 37.5% 
utbanvsrual 
Urban > 10k (less sparse) 21/24 (0) 87.5% 20/24 (0) 83.3% 


Town and fringe/village (less 5 : 
sparse) 3/24 (0) 12.5% 4/24 (0) 16.7% 


Ofsted overalletfecteness 
“WACIrank(Lmostdeprved) | | 
2 (quartile 1 to quartile 2) 7/24 (0) 29.2% 3/24 (0) 12.5% 
iamostepvedy | | | 
(1 most deprived) 
1 (until quartile 1) 10/24 (0) 41.7% 8/24 (0) 33.3% 
2 (quartile 1 to quartile 2) 5/24 (0) 20.8% 4/24 (0) 16.7% 


3 (quartile 2 to quartile 3) 5/24 (0) 20.8% 7/24 (0) 29.2% 
4 (from quartile 3 above) 4/24 (0) 16.7% 5/24 (0) 20.8% 


School-level (continuous) n (missing) Mean n (missing) 


Number of Year 1 pupils 24 (0) 4 
Number of Key Stage 1 pupils 24 (0) 9 


[becigbiefortesscronmeas | | 29 | 24 
winseNfthonasmiemeny | | 38 | em | te 
63 


Percentage of pupils whose first 
language is known or believed 24 (0) 
to be other than English 
Percentage of pupils classified 
as white British ethnic origin 24 (0) ts 
Measures of school 
attainment 
Cohort level Key Stage 1 
% pupils low KS1 attainment 21 (3) 19.2% 20 (4) 19.2% 


% pupils med KS1 attainment 21 (3) 54.8% 20 (4) 58.9% 
% pupils high KS1 attainment 21 (3) 26.0% 20 (4) 21.8% 
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Outcomes and analysis 


The first table in this section (Table 8) shows the raw means for the two treatment groups (the ICT and 
the non-ICT condition) and the two controls groups (control pupils in intervention schools, and control 
pupils in control schools) together with 95% confidence intervals for these means. The first column 
shows the number of observations available at follow-up, that is, the number of pupils for which we do 
have a result in the post-intervention test for each of the outcome variables. The second column 
shows the number of pupils that were part of the randomisation and for which test data is missing at 
follow up. In general, the raw means for groups in treatment schools in the follow-up test are higher 
than the means for the group of pupils in control schools (except for the BPVS test). 


Table 8: Raw means for all outcomes in the different intervention and control groups 


(missing 
n* at follow Raw means (95% Cl) 
up)** 
Outcome variable: PIRA 
ICT 306 54 98.60 (97.13 , 100.06) 
Non-ICT 290 60 99.72 (98.16 , 101.27) 
Control pupils in treatment schools 321 52 98.52 (97.06 , 99.96) 
Control pupils in control schools 1,104 54 97.29 (96.50 , 98.07) 
Outcome variable: DTWRP 
ICT 307 53 109.86 (108.23 , 111.48) 
Non-ICT 288 62 109.82 (108.00 , 111.63) 
Control pupils in treatment schools 319 54 109.85 (108.13 , 111.55) 
Control pupils in control schools 1,102 56 108.59 (107.65 , 109.53) 
Outcome variable: BPVS 
ICT 304 56 89.77 (88.47 , 91.05) 
Non-ICT 284 66 91.33 (89.98 , 92.67) 
Control pupils in treatment schools 315 58 90.62 (89.39 , 91.83) 
Control pupils in control schools 1,094 64 92.94 (92.26 , 93.61) 
Outcome variable: PS 
ICT 307 53 13.42 (13.01 , 13.83) 
Non-ICT 289 61 13.67 (13.23 , 14.11) 
Control pupils in treatment schools 319 54 13.26 (12.85 , 13.66) 
Control pupils in control schools 1,103 55 13.43 (13.22 , 13.64) 
Outcome variable: LK 
ICT 307 53 11.71 (11.56 , 11.85) 
Non-ICT 289 61 11.64 (11.46 , 11.81) 
Control pupils in treatment schools 319 54 11.75 (11.58 , 11.90) 
Control pupils in control schools 1,103 55 11.48 (11.38 , 11.57) 
Outcome variable: LEST 
ICT 307 53 0.83 (0.72 , 0.94) 
Non-ICT 288 62 0.81 (0.69 , 0.92) 
Control pupils in treatment schools 318 55 0.79 (0.67 , 0.90) 
Control pupils in control schools 1,103 55 0.66 (0.59 , 0.72) 


*(n) Number of pupils for which post-treatment test results are available. 
** Missing values calculated from subtracting (n) to the number of pupils in the samples available at randomisation (for more 
information, see subsection Participants in Section 3). 


The impact of the different interventions is summarised in Table 9 for the main outcome variable, the 
Progress in Reading Assessment (PIRA) test. As stated above, this is a test that evaluates the general 
reading ability of pupils. The specifications in the three different columns make use of the maximum 


Education Endowment Foundation 27 


ABRA: Online Reading Support 


number of observations available. All the columns regress the standardised PIRA measure for the 
follow-up test on the ICT and non-ICT treatment dummies, and include dummies for the randomisation 
pairs. As explained in the Participants subsection, this is why the number of schools shown in the 
second to last row of the table falls to 48 schools. Standard errors are clustered at the school level in 
all regressions. For each of the variables, the first row shows the value of the coefficients (that should 
be interpreted in terms of standard deviations (SD) of the outcome variable), the second row shows 
the p-values in parentheses, and the third row shows 95% confidence intervals.*? The control group in 
all regressions consists of all control children (that is, control children in treated schools and control 
children in control schools). Results using control children in control schools only will be shown later in 
the section. 


The first column in Table 9 shows the results of a specification that only controls for the treatment and 
pairing dummies. The second column includes a control for the standardised baseline PIRA test. The 
third column further controls for gender, FSM, and whether the pupil achieved a good level of 
development in the Foundation Stage Profile (FSP GLD). As we progress from columns 1 to 3 there is 
a small increase in the point estimates of the treatment effects and a marked increase in precision. 
Both these changes are more important for the ICT treatment. The results in the most detailed 
specification show the ICT and the non-ICT treatments to have a positive effect on the general reading 
ability of pupils when compared to all control pupils. The ICT treatment increases the general reading 
ability at post-test by almost 14% of a SD whereas the non-ICT treatment increases it by 23% of a SD. 
The last row shows the p-value from testing the null hypothesis that the ICT and non-ICT coefficients 
are equal. The p-value suggests that we can marginally reject the null hypothesis that the effects of 
the two different treatments are equal (at the 10% level). 


This most detailed specification in column 3 is our preferred specification and will be the one used 
throughout the analysis in this section. The section on Methods (in particular, the subsection Analysis) 
compares this final specification with the one stated in the protocol. 


*° Given that the information provided by the p-value and confidence interval is very similar, this third 
row is only added in this first table. 
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4) )) 
Outcome: Penecinenet +FEM, FSM, and FSP 
Standardised PIRA ej] ip) 
ICT 0.074 0.116 0.138** 
(0.321) (0.112) (0.044) 
[-0.074, 0.222] [-0.028, 0.260] [0.004, 0.273] 
0.171* 0.218*** 0.231*** 
Non-ICT (0.052) (0.003) (0.001) 
[-0.001, 0.344] [0.080, 0.356] [0.102, 0.360] 
PIRA lin 0.620*** 0.470*** 
ae 2.00 2.00 
[0.568, 0.671] [0.422, 0.519] 
-0.014 
Female (0.704) 
[-0.085, 0.058] 
-0.175*** 
FSM eligible (SPR14) (0.001) 
[-0.276, -0.074] 
0.521*** 
FSP GLD (0.000) 
[0.426, 0.617] 
-0.491*** -0.193** -0.403*** 
Constant (0.000) (0.016) (0.000) 
[-0.561, -0.420] [-0.349, -0.037] [-0.497, -0.309] 
Pupils 1,952 1,901 1,884 
Missing pupils from 289 340 357 
randomisation 
Schools 48 48 48 
vale ene lease 0.335 0.104 0.103 
(p-value shown) 


Note: Standard errors clustered at the school level. P-values in parentheses (* p < 0.10; ** p < 0.5; *** p < 0.01), 95% 
confidence intervals in brackets. ICT and non-ICT are the treatment dummies. All regressions include pairing dummies (schools 
were paired for randomisation). Both the PIRA outcome variable and PIRA at baseline are standardised to have mean O and SD 
1, so the coefficients are interpreted in terms of SD of the outcome variable. 


As stated in the protocol, Table 10 explores whether there are heterogeneous effects of treatment on 
the main outcome variable according to FSM status, gender, and whether the pupil has a score above 
median attainment on the pre-test. For example, when we consider whether effects are 
heterogeneous by FSM status, we interact each treatment variable with a dummy variable that 
indicates whether the pupil is eligible to receive FSM. All regressions control for the set of controls 
included in the preferred specification (not shown in Table 10). To ease comparability, the first column 
reproduces the main results (i.e. column 3 of Table 9, for the coefficients of interest). The presence of 
heterogeneous effects has also been explored by running separate regressions for each condition 
(FSM and non-FSM pupils; males and females; and above and below median baseline attainment). 
The results remain virtually unchanged. 


The results in the second column of Table 10 show that the impact of the intervention is bigger for 
disadvantaged pupils (as measured by the FSM eligibility indicator). The magnitude of both the ICT 
and non-ICT dummies increase considerably for FSM pupils: ICT and non-ICT treatments increase 
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general reading ability by almost 37% and 40% of a SD for FSM pupils respectively. Non-FSM pupils 
in the non-ICT condition still benefit from the intervention (their results on the PIRA test post-treatment 
are 18% SD higher compared to the pupils in the control groups). However, non-FSM pupils in the ICT 
treatment condition do not seem to benefit from the intervention. The last two rows of column 2 show 
statistically significant differences between the FSM and non-FSM coefficients for both the ICT and 
non-ICT interactions. 


Column 3 repeats the exercise with respect to gender. The results suggest that both males and 
females benefit equally from the ICT intervention. In fact, both coefficients barely differ from the ICT 
coefficient in the main results table (shown here in column 1). Females seem to benefit more from the 
non-ICT intervention than males, although both groups benefit substantially (27% and almost 19% of a 
SD for females and males respectively). However, the results in the last row of Column 3 suggest that 
these coefficients are not statistically different from each other. 


Finally, column 4 examines the interaction between the intervention dummies and the attainment of 
pupils in the PIRA test before treatment. Baseline attainment in the PIRA test is measured with a 
dummy variable that indicates whether the pupil’s baseline test results are above or below the median. 
The results indicate that the non-ICT intervention similarly benefited below- and above-median pupils 
(the p-value shown in the last row of column 4 suggests that these coefficients are not statistically 
different from each other). However, those pupils with below-median prior attainment benefited from 
the ICT intervention whereas this intervention did not affect above-median pupils. In this case, the null 
hypothesis that the coefficients for the two ICT subgroups are equal can be rejected at the 10% level. 
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Table 10: Analysis of heterogeneous effects of the interventions on the main outcome variable 


(1) 4) 6) ) 
Out : Out : 
: eet a tarel| FSM ‘s ween Gender Outcome: Pre-test 
Siandareicee specif | interaction Standardised interaction Standardised PIRA interaction 
PIRA P PIRA 
0.138** 
ee (0.044) 
0.231*** 
NONICT 
eNIe (0.001) 
0.368** 0.139* : 0.055 
ICT*FSM ICT*F | ICT* (> 
re (0.011) emremele (0.085) ere median) (0.383) 
0.081 0.139* 0.215** 
ICT*NOFSM ICT*Mal ICT* (< i 
es (0.217) iene (0.087) cme (0.022) 
0.396*** 0.273*** NONICT* (> 0.235*** 
* * 
NONICT*FSM (0.000) NONICT*Female (0.002) faediany (0.001) 
0.181** 0.187** NONICT* (< 0.230** 
NONICT*NOFSM NONICT*Mal : 
er (0.014) OMCTMe | 751083) median) (0.027) 
Pupils 1884 1884 Pupils 1884 Pupils 1884 
Schools 48 48 Schools 48 Schools 48 
Ho: ICT FSM - Ho: ICT Female Ho: ICT Above - ICT 
ICT NOFSM=0 OCs - ICT Male=0 Ones Below=0 Onee 
Ho: NONICT Ho: NONICT ; 
FSM - NONICT 0.000 Female - 0.421 poe aeaaeaed 0.964 
NOFSM=0 NONICT Male=0 . 


Note: Standard errors clustered at the school level. P-values in parentheses (* p<0.10; ** p<0.5; *** p<0.01). ICT and NONICT 
are the treatment dummies. All regressions include pairing dummies (schools were paired for randomisation) and the controls of 
the main specification: PIRA baseline (std), Female, FSM eligible, and FSP GLD. The PIRA outcome variable is standardised to 
have mean 0 and SD 1. ‘> median’ and ‘< median’ stand for above and below median pre-treatment PIRA attainment. 


In Table 11 we address whether there have been spillover effects from treatment pupils to control 
pupils in treatment schools. The protocol specified that we would run a regression where the control 
group would be formed by pupils in the same schools, and a second regression where the control 
group would be formed by pupils in control schools only. We could then check whether the treatment 
effects were larger in the second regression than in the first one. This would be indicative of spillover 
effects. Although we do this, we also run a regression where we simply include an additional dummy 
variable for pupils that are in the control group of treatment schools (denoted by CT). The advantage 
of this approach is that it produces similar insights without having to reduce the number of clusters 
(which is necessary when running regressions for treatment schools only: 24 schools). In Table 11, 
column 1 replicates our preferred specification for the main outcome (as shown in column 3 of Table 
9, showing coefficients on the treatments). In column 2 we include a dummy variable (CT) indicating 
whether pupils are in the control condition within treatment schools. Column 3 uses only the 
observations related to pupils in treatment schools and column 4 shows results where we exclude 
pupils assigned to the control condition within treatment schools. The coefficient CT in column 2 
shows that control pupils in treatment schools did significantly better at post-test than control pupils in 
control schools. In fact, they did just as well as those exposed to the ICT treatment. However, both 
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groups are still out-performed by those exposed to the non-ICT treatment. A very similar picture 
emerges in column 3. Column 4 presents results where we exclude pupils assigned to the control 
condition within treatment schools (in other words, the control group is only comprised of pupils in 
control schools). The magnitudes of both the coefficients on the ICT and non-ICT treatments increase 
considerably in this case. Along with the results in columns 2 and 3, these findings are consistent with 
the presence of spillover effects from treatment pupils to control pupils in treatment schools, although 
the mechanism for these spillovers is not clear. A potential explanation for spillover effects is that 
teaching assistants interacted with pupils on other occasions. They were not only employed to 
implement this experiment. Potentially, the training enhanced their ability to help pupils at other times. 
This would have affected the control group as well as the treatment groups. 
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Table 11: Spillover effects 


elute e oltye 
e e e e e e e 
e e i OH PIRA el= ele ele ele 
ee 0.138** 0.179** 0.013 0.186** 
(0.044) (0.027) (0.792) (0.025) 
0.231*** 0.272*** 0.080* 0.284*** 
NONICT (0.001) (0.001) (0.092) (0.001) 
0.167** 
et (0.028) 
0.470*** 0.469*** 0.523*** 0.478*** 
PIRA basel 
pereine st) (0.000) (0.000) (0.000) (0.000) 
ne 0.014 -0.011 0.078 -0.044 
(0.704) (0.769) (0.123) (0.272) 
- -0.175** | 0.178% -0.168*** -0.161*** 
POMeloiie ornia (0.001) (0.001) (0.001) (0.006) 
0.521*** 0.531*** 0.428*** 0.517*** 
FSF_GLD (0.000) (0.000) (0.000) (0.000) 
Constant -0.403** | -0,437** 0.243 -0.407*** 
(0.000) (0.000) (0.000) (0.000) 
Pupils 1,884 1,884 870 1,576 
Schools 48 48 24 48 
P-values 
7 0.102 
He We ONC 2 0.103 0.820 0.244 0.092 
Ho: ICT-CT= 0 oe 
Ho: NONICT-CT=0 


Note: Standard errors clustered at the school level. P-values in parentheses (* p < 0.10; ** p < 0.5; *** p < 0.01). ICT and 
NONICT are the treatment dummies. CT is a dummy that equals 1 for pupils in the control group of treatment schools. All 
regressions include pairing dummies (schools were paired for randomisation). Both the PIRA outcome variable and PIRA at 
baseline are standardised to have mean 0 and SD 1. 


Finally, Table 12 shows the results using the preferred specification (that is, controlling for pairing 
dummies, gender, FSM eligibility and FSP GLD) for all outcome variables. The number of 
observations differs between columns because we have included all available observations for each 
outcome variable. However, if we restrict the observations in these regressions to include those pupils 
for which we observe all six outcome variables, the results are almost the same. As in Table 9, the 
control group in all regressions in this table consists of all control children (those in treatment schools 
and those in control schools). 


Of the five secondary measures analysed, the largest impact of the intervention is shown for the Letter 
Sound Test (LEST in column 6)—a test that assesses a person’s ability to sound out single letters and 
letter combinations. This is directly related to one of the four modules of ABRA—the letters and 
sounds module teaches a subset of the letter combination sounds tested in LEST. Both the ICT and 
non-ICT interventions have a positive and sizable impact on the LEST outcome (almost 22% and 17% 
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respectively, although the two coefficients do not significantly differ from each other). The ICT and 
non-ICT treatments are also found to have a positive effect on the DTWRP test (although only the ICT 
treatment has a significant effect at conventional significance levels). Pupils in the non-ICT condition 
also seem to benefit over controls in terms of their ability to perform phoneme segmentation as 
measured by the subtest of PIPA (column 4). Finally, the interventions did not seem to affect 
attainment in the Letter sound subtest of PIPA (column 5), and the results suggest that the ICT 
intervention had a negative effect on a child’s receptive vocabulary as measured by the BPVS test 
(column 2). This will be explored further in future research. 


Table 12: All primary and secondary outcomes 


(1) (2) &)] ) iC) (3) 

PIRA DTWRP- BPVS_ PSS LK LEST 

= 0.138  0.107* | -0.132 | 0.024 0.090  0.219*** 

(0.044) (0.062) | (0.015) | (0.678) (0.172) | (0.001) 

fae 0.231** | 0.096 -0.058  0.096* 0.068 0.168" 

ai (0.001) | (0.140) | (0.200) | (0.094) | (0.345) (0.035) 
arene 0.470** | 0.690" 0.674*** 0.004 -—0.423"** 0.537" 
we P (0.000) (0.000) (0.000) (0.883) (0.000) — (0.000) 
oem 0.014 -0.067** | -0.063** | -0.020 0.026 | -0.010 
(0.704) | (0.046) | (0.022) | (0.670) (0.633) | (0.819) 

FSM eligible -0.175*** 0.099  -0.028  -0.107*  -0.163*" — -0.109* 
(SPR14) (0.001) | (0.016) | (0.481) | (0.040) (0.024) — (0.066) 
es 0.521" | 0.284" 9.219%" 0.323"  0.081* 0.351" 
(0.000) | (0.000) | (0.000) | (0.000) (0.087) — (0.000) 
rem -0.403** | -0.312"* | 0.300%" | -0.276 | 0.069 | -0.563"* 
oe (0.000) | (0.000) | (0.000) | (0.036) | (0.620) | (0.000) 

Pupils 1,884 1,881 1,857 1,912 1,912 1,899 

Schools 48 48 48 48 48 48 
Heer NOME | aaoe 0.879 0.224 0.167 0.739 0.508 
(p-value shown) 


Note: Standard errors clustered at the school level. P-values in parentheses (* p < 0.10; ** p < 0.5; *** p < 0.01). ICT and non- 
ICT are the treatment dummies. All regressions include pairing dummies (schools were paired for randomisation). All the 
outcome variables and the outcome variables at baseline are standardised to have mean 0 and SD 1. In each column, the 
variable standardized values of each pre-test correspond to the standardised value of the pre-test for that particular outcome 
variable. For instance, in column 1 this is the standardised value of PIRA (DTWRP) at baseline. 


Cost 


The interventions cost the schools nothing in this trial. Here we report the costs associated with 
implementing each intervention had the EEF had not funded it. 


The software for the ICT intervention is available free on the internet. The activities for the non-ICT 
intervention mirrored those in the ICT intervention. The 20-week lesson plan for both the ICT and non- 
ICT intervention were supplied free of charge to the schools. 


We used professorial level academics to run the training for the TAs who participated. This was 
funded by the EEF at a rate of £750 per day for the non-ICT intervention and £500 per day for the ICT 
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intervention. However, the training would be delivered in future by academics on a lower pay grade 
(£300 per day).7° 


Although the interventions themselves were delivered without charge, there were some costs 
associated with printing the 20-week programme for each intervention: £3.50 per manual for the ICT 
intervention and £22 per manual for the non-ICT version. 


The implementation team trained each TA for 1.5 days prior to involvement in groups of 12 to 13. On 
average, each TA received approximately 0.6 days of further support from the project implementation 
team (a mix of in-person, phone, and email support) at a cost of £300 per day. Forty-nine TAs were 
trained to deliver the intervention to 710 children. 


In addition, to use the ICT intervention, schools need to have access to a computer and the internet. 
This was already in place in all the schools that participated. 


Total cost of ICT intervention in first year (following year’s cost = £0): 


Cost of training 25 TAs = 1 group of 12 TAs at £500 for 1.5 days = £750; 1 group of 13 TAs at £500 for 
1.5 days = £750; 

Total training cost = £1500 

Cost of support for 25 TAs = £300 x 0.6 x 25 = £4500 

Cost of 25 manuals = £3.50 x 25 = £87.50 

Total cost = £1500 + £4500 + £87.50 = £6087.50 for 360 children = £16.91 per child 


Total cost of non-ICT intervention in first year (following year’s cost = £0): 


Cost of training 24 TAs = 1 group of 12 TAs at £750 for 1.5 days = £1125; 1 group of 12 TAs at £750 
for 1.5 days = £1125; 

Total training cost = £2250 

Cost of support for 24 TAs = £300 x 0.6 (of a day) x 24 = £4320 

Cost of 24 manuals = £22 x 24 = £528 

Total cost = £2250 + £4320 + £528 = £7098 for 350 children = £20.28 per child 


The cost of delivery based on a post-doctoral academic delivering the training (a research fellow) 
would reduce the cost of the ICT intervention to £15.24 per child, and £16.42 per child for the non-ICT 
condition. 


Total cost of ICT intervention in first year (following year’s cost = £0) based on a post-doc 
delivery: 


Cost of training 25 TAs = 1 group of 12 TAs at £300 for 1.5 days = £450; 1 group of 13 TAs at £300 for 
1.5 days = £450; 

Total training cost = £900 

Cost of support for 25 TAs = £300 x 0.6 x 25 = £4500 

Cost of 25 manuals = £3.50 x 25 = £87.50 

Total cost = £900 + £4500 + £87.50 = £5487.50 for 360 children = £15.24 per child 


Total cost of non-ICT intervention in first year (following year’s cost = £0) based on a post-doc 
delivery: 


° These price differences are due to the delivery organisation and not reflective of different levels of 
qualification 
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Cost of training 24 TAs = 1 group of 12 TAs at £300 for 1.5 days = £450; 1 group of 14 TAs at £300 for 
1.5 days = £450; 

Total training cost = £900 

Cost of support for 24 TAs = £300 x 0.6 (of a day) x 24 = £4320 

Cost of 24 manuals = £22 x 24 = £528 

Total cost = £900 + £4320 + £528 = £5748 for 350 children = £16.42 per child 


Costs could be reduced further by supplying the 20-week programme to schools in electronic format. 
This would bring it down to £15 per child for the ICT and £14.91 for the non-ICT intervention. 


Addendum 1: 


In this trial, schools were paid for TAs to be able to attend training (they were paid £128 each). A total 
of £128 x 49 TAs = £6272. For 710 children, adds £8.83 per child. Their travel costs were also paid, at 
an estimate to be £25 per TA: £25 x 49 TAs = £1225. This adds £1.73 per child. Thus, the total cost 
for expenses to cover training are equal to £8.83 + £1.73 = £10.56 per child. 

If we consider the previous estimates with the 20-week programmes in electronic format, the cost per 
child would be £25.56 (ICT) and £25.47 (non-ICT). 

NOTE: Following the EEF guidance, the impact on staff time should be reported separately in units of 
time. Given that most schools did not appoint new TAs or extended hours of existing TAs, we have not 
included a salary cost for the TAs in the above estimate. In terms of time, TAs are calculated to have 
employed 4160 hours for 710 children. This amounts to 5.85 TA hours per child per year. If we further 
consider that TAs needed about 5 minutes per day to prepare the sessions, then the estimate of TA 
hours per child per year increases to 6.32 hours. 
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4. Process evaluation 


The purpose of the process evaluation was to assess the fidelity of the interventions and to 
help explain the impact of the ABRA programme as measured by the RCT. We aimed to 
achieve this largely through assessing the practices and experiences of teaching assistants in 
participating schools, and through observation of the two interventions—ICT and non-ICT— 
focusing on teaching and learning processes, particularly pupil response and engagement. 
We interviewed staff and observed sessions in six selected intervention schools, following up 
these visits close to the end of the intervention. We also observed introductory and training 
sessions. In addition we used quantitative data collected by the project implementation team, 
including their assessments of fidelity and quality of delivery. 


We have structured the findings from the process evaluation according to the main criteria 
used by the EEF: implementation, outcomes, and fidelity. We have also assessed control 
group activity based on data collected by ourselves and by the project implementation team. 


Implementation 


Background information on TAs in the active schools was collected through a survey carried 
out by the project implementation team at Coventry University. This found that TAs had a 
range of qualifications for the job, commonly NVQ levels 2 or 3, Higher Level Teaching 
Assistant (HLTA) qualifications and, in two cases, an early years’ degree. Most had 
experience of working across year groups with years of service ranging from a few years to 
more than 30. TAs main duties varied widely. They included standard TA duties such as 
supporting class teachers, playground duties, and first aid. Teaching phonics, guided reading, 
and maths interventions were also part of the role for many. 


The TA survey found that almost all had been involved in delivering literacy programmes as 
part of standard literacy provision. Specific programmes included ‘Letters and Sounds’, ‘Jolly 
Phonics’, ‘Phonics Bug’ and ‘Read, Write Inc’. All of the TAs interviewed had assisted children 
with literacy before being involved in the ABRA programme. Many, but not all, had done 
group work and in some cases this was for teaching phonics. For TAs, the main difference 
between their previous experience and ABRA was its status as an ‘intervention’ which 
required delivery in specified ways. But this in itself was not new to most TAs: many had 
previous experience of interventions and felt this equipped them well for the project. Despite 
their previous experience of literacy work, the programme itself in either the ICT or non-ICT 
versions was different to anything that most had used before. The main difference was in the 
complete and packaged nature of the intervention and the requirement to follow it closely, 
including through time allocation of components within delivery. As one TA explained: 


‘I’ve never done it quite so, two minute, three minute, five minute. 
I’ve never done it quite so rigid as that before’ (non-ICT TA). 


The pace of the sessions was also new to many TAs who had been accustomed to more 
leisurely activities with individuals or groups. The group work aspect (where support also 
distinguished it from programmes such as Better Reading Partnership, used in one of the 
schools) constituted another novel feature for many of them: TAs were more accustomed to 
delivering support on a one-to-one or one-to-two basis than to groups of pupils. 


The ICT intervention was seen as more novel than the non-ICT package. The TA survey 
found that most said they had used ICT for literacy and numeracy. However, those who we 
interviewed said that while they had used programmes with pupils before it had been in quite 
a limited way and rarely for phonics teaching. 
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Training and preparation 


TAs had first known about the intervention shortly before the training session. Most said they 
had been interested in ABRA when told they would be taking part. However, some had been 
concerned about how they were going to fit an additional activity into their working day. 


Following an introductory conference in May 2014 attended largely by senior leaders, the 
project team organised 1% days of training for teaching assistants before the start of the 
intervention. These were delivered during autumn 2014 (for cohort 1) and autumn 2015 (for 
cohort 2). During the full day of training TAs were introduced to the interventions and trained 
how to use them. The half-day event took place two weeks later after they had had time to try 
out the activities and feedback on their own experience of delivering the intervention activities. 


At the full day of training in October 2014, TAs were introduced to teaching reading as well as 
trained to use the 20-week intervention, with hands-on practice and feedback from instructors. 
The training was interactive, and TAs were encouraged to ask questions throughout the day. 
The ICT and non-ICT interventions were designed to be as similar as possible, focusing on 
effective small group literacy strategies and the specific activities used throughout the 
programme. Training in the ICT intervention was delivered by Robert Savage, Associate 
Professor at Canada’s McGill University, part of the team responsible for the development of 
ABRA. Training in the non-ICT intervention was delivered by Morag Stuart, Emeritus 
Professor in the Psychology of Reading at UCL’s Institute of Education. Topics covered by 
the training are shown in Figure 2. 


Figure 2: ABRA training 


Introduction to teaching reading: 

¢ how to use the interventions as a tool to teach children skills to maximise their reading 
outcomes in the broadest sense 

¢ basic reading skills—decoding, fluency, and comprehension 

¢ why the basic reading skills are important to reading outcomes 

¢ teaching multi-ability groups 

* managing behaviour in groups and setting group rules 


The training on the 20-week intervention: 

¢ the length and number of sessions to deliver 

* the aims of each of the activities and how to deliver them 

¢ how to keep records of pupils’ progress and attendance 

¢ how to set (and track) the level of each activity to match that of the pupils 

¢ how to access help on each of the activities (in print for non-ICT, on the laptop 
for ICT) 

¢ how to access (just in time) support during delivery of the intervention 


Hands-on practice: 

¢ free time to explore the activities and resources 

* group time to deliver/role play individual activities 

* group time to deliver/role play a whole session (i.e. 3 or 4 activities) 

¢ structured sessions to feed back experience of delivering sessions and activities 
¢ structured sessions to troubleshoot and share good practice 


At the end of the full training day, TAs were asked to go away and practice some of the 
sessions with children not in the study, and be ready to feed back on their experiences at the 
half day training two weeks later. They were given contact details to access help with the 
intervention delivery (email and phone contacts). At the follow-up half day of training, TAs fed 
back on their experiences of delivering sessions, shared good practice, and discussed any 
problems they had encountered—and solved these with each other and the instructor. Any 
questions they had about delivering the interventions were answered during the session. Any 
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extra time was used to practice activities they had not yet practiced. Once delivery of the 
interventions had started, Coventry University provided support to the TAs, through a 
combination of visits, email, or phone depending on the kind of support needed. 


As we explain later, the initial training emphasised the importance of fidelity and in particular 
the need to follow lesson plans and keep to timings. The training also included guidance on 
managing group work, for example by establishing ground rules. 


We asked teaching assistants about their experiences of the training during our visits to 
schools in January 2015 and also analysed training evaluation data collected by the project 
team. We also attended training days 1, 2 and 4 where TAs were trained in separate groups 
in the ICT and non-ICT interventions. 


Data collected through the interviews and evaluation forms shows TAs consistently rated the 
training highly with almost all in both the ICT and non-ICT intervention training strongly 
agreeing that the goals of the training had been met and that they were satisfied with their 
increased understanding of the topic. Those in the ICT group were more likely to express this 
than those in the non-ICT training. The evaluation forms asked TAs their level of agreement 
with the following statements: 


e The goals of the training were clearly defined. 

e The topics covered were relevant. 

e The introduction to each session stated the objectives clearly. 

e There was sufficient opportunity for interactive participation. 

e The training was too technical and difficult to understand. 

e The training experience will be useful in my work. 

e Most of my questions were answered during the training. 

e The materials were pitched at the right level. 

e The materials for the training were helpful. 

e The schedule for the training provided sufficient time to cover all of the proposed 
activities. 

e The handouts provided were helpful. 


Almost all TAs in both ICT and non-ICT groups agreed or strongly agreed with these 
statements, with the exception of ‘the training was too technical and difficult to understand’ 
where the TAs felt this was not the case. TAs in the ICT group were more likely than those in 
the non-ICT group to agree strongly with the statements, with around half in the ICT group 
agreeing strongly with most of them. These high levels of agreement do not indicate that the 
training was easy. TAs interviewed delivering the ICT based intervention felt there was a lot to 
learn, particularly those who saw themselves as not technically-inclined. Some said that, on 
reflection, they would have liked a little more time to explore the different parts of the 
programme, but that they had practised enough to be able to implement it in school. TAs 
interviewed during the school visits felt the training had prepared them well for the 
intervention. They were also reassured that they could contact the Coventry University team 
when they needed to, and some said they had done this. Examples of where this had been 
necessary included whether to remove a child because his behaviour was preventing other 
group members from learning. The Coventry team had also provided additional equipment 
such as laptop speakers and replacement timers. 


Organisational issues 


For schools, the main organisational issue involved allocating TAs and their time to the 
intervention. TAs on the project were largely drawn from those working with Year 1 pupils, 
although some from other years were included where additional numbers were required. All 
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equipment was provided by the project. Schools then had to allocate time in the school day to 
deliver the programme. Schools were asked to deliver it during literacy-based lessons but not 
core literacy instruction, including phonics work. This is because the intervention was 
designed as an additional programme, not a substitute for normal classroom delivery of 
literacy. Visits by the project team enforced this feature of the programme, correcting schools 
which were timing sessions to coincide with core literacy work. The TA survey found that no 
school was taking pupils from core literacy classes and most delivered the intervention during 
general literacy-based sessions, such as guided reading or topic work. This was confirmed by 
school visits by the external evaluator: sessions were reported to take place largely during 
non-core literacy time, although was sometimes also during break time or PE (physical 
education). In some schools the intervention was timed flexibly so that pupils did not regularly 
miss the same lesson, particularly where this was literacy. 


Support from schools and the project team 


While some schools faced some challenges, particularly with space to deliver the intervention 
(see below), TAs said that they had been fully supported by senior leaders within their 
schools, including covering their duties, as well as by the project team at Coventry University 
and had experienced few problems. The TA survey found that most felt very well supported 
by their school, with 31 of 37 saying they felt very or somewhat supported; 25 of 37 TAs said 
they felt very supported by the project team at Coventry University and the rest said they felt 
somewhat supported. Despite this support, a small number of TAs responding to the survey 
indicated that the project impacted on their workload and they had to spend additional time on 
preparation and writing up. At the follow-up stage there were reports that TAs felt that the 
demands of the project impacted on the time they could spend supporting other children and 
class teachers. 


Technological and skills issues 


The survey carried out by the project implementation team asked TAs whether their use of the 
intervention was ever affected negatively by technological problems and limitations. Most TAs 
delivering the ICT intervention did report problems but these appeared to be minor and 
occasional, for example a screen freezing. No TA reported frequent technological problems. 
Very few TAs said their use of the intervention was negatively affected by their skills. Of these 
few, their reasons were varied, including lack of preparation time and confidence that they 
were delivering it correctly. 


Practicalities 


There were a number of practical issues for TAs: taking pupils to and from sessions, space 
within the school, and the short length of the sessions. The survey found that 24 out of 37 
TAs said the intervention was sometimes negatively affected by school based problems or 
limitations, such as time and location. Six TAs said there were frequent such limitations and 
problems. Only six said these never occurred. Problems included space and clashes with 
other school activities. TAs interviewed confirmed that space was one of the main practical 
issues encountered by both the ICT and non-ICT interventions. Sessions were delivered in a 
range of locations within the schools visited, including spare rooms, classrooms alongside 
lessons, and in corridors. As one TA explained, this was sometimes time-consuming: 


‘You walk through the school. Everywhere is occupied. It all takes 
time, especially if you have to do three groups a day.’ 


Some of these settings were distracting for pupils and made it difficult for them to hear the 
ICT programme voiceover. 
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TAs interviewed also reported practical issues with the need to take pupils from class to the 
activity. This was reported to be sometimes time-consuming, for example where pupils were 
collected from PE and needed to change their trainers for shoes (not a fast process for a five 
year-old). Teachers were reported to be sometimes concerned at the disruption to their 
lessons. 


Length of sessions and group size 


The length of sessions, at 15 minutes, was seen by some TAs as too short. For the ICT 
programme in particular, the 15 minutes was eroded by time required to settle pupils down, 
log on, and refresh pupils about the previous session (see later). At the same time, TAs 
delivering both interventions felt that much longer would challenge pupils’ attention. One view 
was that time constraints made the ICT programme less beneficial to less able children who 
need more time to work through the activities before moving onwards. Some TAs delivering 
the non-ICT intervention found it challenging to keep to the timings within sessions, but again 
became accustomed to their structure. TAs in both the ICT and non-ICT interventions were 
instructed to move between levels only when the group had reached a correct response rate 
of 80%. Some TAs had been a little concerned about this requirement, feeling it might be 
difficult to decide. In practice, they found this easy to follow. 


The size of pupil groups, at between three and five, was seen as about right, allowing each 
child to have an input but to also learn from others in the group. For the ICT intervention a 
smaller group allowed pupils to interact more directly by using the mouse themselves. In 
larger sessions the TA tended to control the mouse for most of the sessions observed. More 
than five was seen as too many for the ICT intervention but less so for the non-ICT sessions. 
While the project allocated pupils to the two interventions or the control, TAs were able to 
allocate pupils into groups within the intervention, and did this largely according to ability 
level. This was seen to work better than mixed ability groups, particularly for the ICT 
intervention. 


Attractiveness of the intervention to stakeholders 


TAs found the ABRA programme attractive in both its ICT and non-ICT forms. They 
welcomed the structured nature of the programme and the full set of resources provided for 
both interventions. TAs delivering the non-ICT intervention liked the paper resources, which 
include a series of storybooks. TAs delivering the ICT version liked the programme, its 
graphics and activities. A number commented that, before using the programme, they usually 
had to source at least some resources for themselves and liked these being brought together 
in a complete package. One advantage of the non-ICT intervention was its perceived greater 
adaptability for different ability levels. 


TAs on both interventions enjoyed delivering small group teaching. They liked improving their 
literacy teaching skills and those in the ICT group felt they had improved their navigation and 
mouse skills. They also liked the programme itself although, in time, some became somewhat 
bored with the repetition it inevitably involved. TAs delivering the non-ICT intervention also 
liked the resources, in particular their completeness as a package and the absence of 
photocopied worksheets. 


As a TA-led intervention, we were interested to know participants’ views on this aspect of the 
design. Mixed views were expressed on whether pupils responded differently to a TA-led 
intervention than if it had been delivered by a teacher. Some TAs felt that pupils were more 
likely to take risks in their responses with a TA, particularly within a small group. One 
described the atmosphere of a TA-led group as more ‘homely’ or nurturing because of TAs’ 
pastoral role. However, others felt that it was small group teaching which changed pupil 
behaviour, that pupils were not able to draw a distinction between TAs and teachers and that 
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either could form a relationship of trust. As one TA delivering the non-ICT intervention 
explained: 


‘If someone else is answering the question, they’ve not got to 
answer it so will just sit there and zone out sort of thing, but when 
you're asking them it’s more one-to-one, they’re actually having to 
get involved. They’re concentrating more because they have to be 
able to answer the question.’ 


The positive response of pupils was an important motivating factor for TAs. This was 
particularly strong for the ICT intervention which some TAs said appealed strongly to pupils. 
Their belief that the intervention was making a difference to pupils’ literacy skills was also an 
important factor in their commitment to, and interest in, the project. 


Preparation time was an issue for some TAs interviewed. During the training, TAs were told 
that little planning time was required for each session. However, TAs found that they did in 
fact need some time to prepare, but usually no more than five or ten minutes. Both 
interventions required some set up time and the ICT version required additional log-in time. 
Some schools had allocated such time, for example a TA with three non-ICT groups had 15 
minutes a day for preparation. In another school, TAs were simply re-allocated time assigned 
for guided reading preparation. Despite these allowances, the intervention presented some 
challenges to the time management skills of some TAs. 


A potentially less attractive side of the intervention was the requirement on TAs to keep 
records of each session, indicating the level that pupils had reached. They had been 
concerned that record-keeping would be onerous but generally found it straightforward and 
even useful for subsequent sessions, reducing planning time. Some TAs took only rough 
notes at the end of sessions and wrote them up more fully later and did not see this as 
problematic or excessively time consuming. Some also made their own additional notes, for 
example recording high frequency words identified by pupils. 


Pupil response 


As stated above, TAs were motivated by the positive response of pupils. They reported that 
pupils enjoyed the sessions, both the ICT and non-ICT versions. Pupils were reported to look 
forward to going and to enjoying being taught in a small group. As one TA delivering the non- 
ICT version explained: 


‘They love it, not because of this particular session but because 
they get the group and one-to-one attention. ’ 


One TA delivering the non-ICT intervention described how children skipped across the 
playground in haste to get to the session. Pupils’ enjoyment of the intervention was explained 
with reference to a number of factors: the small group format, the interactive nature of the 
sessions, the humour and the ‘game’ aspect, and more generally the appeal of the resources. 


Some TAs felt that pupils enjoyed sessions more when groups were small, for example only 
three pupils, and where each pupil could participate. Higher-ability pupils were seen to 
particularly welcome being involved, since interventions are usually targeted at less able 
pupils. A TA explained: 


‘Lots of the children who are getting to do it would never be in an 
intervention, it’s a novelty. They say that splitting the children up in 
abilities of their peers makes for more effective delivery.’ 


These pupils were found to enjoy the interactive nature of both the ICT and non-ICT sessions 
and the opportunity to demonstrate their knowledge and skills. As another TA explained: 
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‘One of the questions in the Frogs and the Well is “have you seen a 
pond like this before?”. The lowers say “Yes, I’ve seen a pond in 
Cannon Hill Park”. The higher-ability kids say “Yes on TV I saw 
this”, and they have all this knowledge that they are desperate to 
share. | think within the class they don’t get to use these voices and 
to talk about what they know. So this | think is really important, for 
the lowers and the higher. | think it’s got something for both, and the 
middles just the same.’ 


Pupils clearly enjoyed the ICT based sessions. They engaged in the programme, characters, 
language and stories. They enjoyed the humour of the programme, for example recognising 
incongruous words and phrases, such as ‘went down to the river to get a hamburger’. Pupils 
also enjoyed shouting out words as they appeared on the screen. In a number of the groups 
pupils expressed an enthusiastic preference for particular activities, for example changing a 
letter to change the word. The pace of the ICT lessons was fast, led by the programme. This 
kept most pupils engaged and could give an advantage to higher achievers and faster 
learners. However, it was common for pupils to work as a group rather than for individual 
pupils to dominate response activities. TAs commented on the game format of the ICT 
programme which they felt helped pupils to learn while playing. As TAs from two different 
schools explained: 


‘it's more of a game situation so they don’t have to think as much... 
They find it easier to do something if it’s a bit more fun.’ 


‘When they're coming out of the class it feels like this is going to be 
a bit of fun, let’s crowd around the computer, let’s just have a bit of a 
game. Sometimes they'll say, “If we go back to the class we're 
learning in there. This isn’t learning, this is games” and they don’t 
even realise that what they’re doing is integral learning for them.’ 


This also had its downside in pupils sometimes not understanding why they couldn’t choose 
which ‘games’ to play. 


Pupils were also reported to find the non-ICT resources attractive. They liked the books and 
stories and receiving stickers at the end of the session. As one TA explained, in contrast to 
the usual photocopied sheets that pupils work with: 


‘The books are coloured, so for them to open coloured books is 
different. You don’t get that in class. Normally for phonics you have 
paper worksheets.’ 


However, the pace of the non-ICT sessions was reported to be slower with pupils expressing 
boredom with reading the same book for a number of weeks. 


A few drawbacks to the ICT intervention were identified. First, the Canadian accent and, toa 
lesser extent, vocabulary, was a minor distraction for both TAs and pupils. One example of 
where this presented difficulty was in changing word meanings through letter change: ‘ten’ to 
‘tan’ is less apparent to an English ear when spoken in a Canadian accent. Some TAs 
highlighted any differences to pupils and therefore made this into a learning point. For 
example, one TA told pupils a few times ‘I would say that differently’. Some words familiar in 
Canada were not understood, for example ‘recess’, ‘dike’ and ‘blimp’. Children in one group 
mistook ‘Caribou’ for ‘car boot’. This led to a short discussion on whether a caribou is a 
reindeer or a moose. Some commonly used English words were also not always understood. 
For example, one group of pupils did not understand some common terms such as ‘current’ in 
the context of a river, or knew what a wig was. Other words or phrases were just seen as too 
advanced, for example ‘indestructible igloo’ and ‘Ursula’s upturned umbrella’. Some TAs 
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commented that as well as including Canadian vocabulary, the programme had a middle 
class bias, again apparent in the language (although examples were not given). 


A second drawback to the computer-based intervention concerned the ICT skills of some 
pupils. It was apparent that the mouse skills of some pupils were not well developed. In some 
cases interaction with the programme was largely TA-led. This appeared to be done in some 
cases to ensure that the pace of the lesson was kept up so that activities were completed on 
schedule. 


Pupil engagement was good during the sessions observed for the external evaluation. 
However, TAs had to work harder to engage pupils, to keep their attention, and to ensure a 
steady pace. One TA used a teddy bear to add interest to the session (teddy was told a word 
and pupils were asked what letter it begins with). Some pupils and groups appeared to find 
the non-ICT intervention somewhat repetitive. Pupils of higher ability in one group complained 
they found the decoding activities ‘too easy’. For this reason, TAs in another school had 
changed the balance of activities with higher-ability pupils spending more time on 
comprehension than on decoding and blending. 


Our observations found pupils engaging more fully with the ICT programme than the non-ICT 
intervention. While pupils liked the books and activities, these had less appeal than the ICT 
programme. At the same time, some pupils were very positive about the non-ICT sessions, 
with one group going as far as describing the sessions as ‘fantastic’ to the NIESR evaluator. 
In that particular group, of higher-ability pupils, the TA felt the activities enabled pupils to learn 
from each other’s responses. 


We referred earlier to the space problem experienced by some schools. Pupil response to 
both interventions depended in part on location of the sessions. Some sessions were held in 
classrooms where other activity was in common areas or corridors. TAs had to work harder to 
keep pupils’ attention in such circumstances, and faced more challenges with the non-ICT 
sessions. TAS managed these challenges very well. 


Fidelity 


The importance of implementing the project faithfully was impressed on TAs during the 
training. Particular emphasis was placed on not sharing information about the training or 
project materials with TA colleagues delivering the different intervention. TAs also understood 
from the training that the intervention had to be delivered as specified. This meant: 


e keeping to timings; 

e moving on to higher levels only when pupils were ready, measured at having reached 
80% accuracy in responses; and 

e keeping records. 


The Coventry University project team put a range of fidelity control measures in place. These 
included requiring TAs to record progress made in each session through completing tracking 
sheets. The project team also kept in regular contact with schools and TAs and visited each 
school to observe all TAs involved in delivering the project. The purpose of this was to check 
that both the ICT and non-ICT interventions were being delivered at the level of expected 
quality. 


Our visits found that some TAs had been concerned about these requirements but in practice 
found few difficulties. As one explained: 
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‘When | first saw [the record sheets] on the training, | was like “Oh 
God, paper work, how am | going to deal with it?” But they’re quite 
easy’ (non-ICT). 


TAs took the requirements of the project very seriously, with very few saying that they 
deviated from the guidance in any way. Those who did made very small adaptations, for 
example writing notes during the session and writing them up later. Responses to the survey 
carried out by the project implementation team indicate that none delivered it in any setting 
other than a small group. 


TAs were also asked to keep the ICT and non-ICT interventions distinct by not sharing 
information about the content and delivery of the two programmes. The project team survey 
found that most TAs did not see the other intervention being delivered, though six did. 
However, the level of awareness of the actual content of the alternative intervention was low: 
most simply said they knew that one involved ICT and the other paper-based resources. 
Those who appeared to have more of an idea of differences, for example that the ICT 
intervention was ‘games focused’ were not confident that they really knew. It is therefore very 
unlikely that either approach was contaminated by the other in terms of content, delivery, or 
anything else. 


Scheduling and delivery 


TAs were advised at the training stage to follow the programmes to the letter and to tell the 
project team if they were not able to keep to the lesson plans. The project implementation 
team did not wait for TAs to approach them with any concerns or questions, although some 
TAs did make such contact and in some cases quite regularly. The Coventry team very 
actively engaged with TAs by making regular contact. This would seem to be an effective way 
of ensuring that an intervention is delivered with fidelity. It may be particularly important for 
project teams to take this proactive approach in TA-led interventions given that TAs are likely 
to have less time than teachers to prepare or follow-up on project-based activities (Blatchford 
et al., 2012). 


Visits by both the project implementation team and by external evaluators found schools 
implementing ABRA according to the project design. This included keeping attendance 
records and keeping to the lesson plans and activity timings. TAs were advised not to 
advance to the next level of activity until the group had scored 80%. Those interviewed during 
the evaluation visits said they had followed this guidance. TAs also said they recorded 
progress made in each session, despite initially expressing concerns about this. 


TAs responding to the project implementation team survey reported spending varying periods 
of time delivering the sessions. Fourteen said they spent 15 minutes, as specified in the 
training and guidance; eighteen said they spent between 15 and 20 minutes. Only three said 
they spent more than 20 minutes but that this included getting children ready, which others 
reported was time-consuming. 


The tracking sheets required TAs to show that a session had taken place and whether any 
pupils in the group were absent. Two members of the project implementation team scored 
these sheets independently for quality, compared results, and arrived at a final score. 
Records were collected from 49 TAs. The exercise found that 34 out of 49 TAs provided 
complete records of their sessions. Of those who did not, only four TAs were scored at five 
(out of ten) or below, indicating consistently poor record-keeping rather than some missing 
records. The tracking sheets also asked TAs to record evidence that they were moving pupils 
appropriately through the levels, moving on once they achieved 80% accuracy in response. 
The project team found this somewhat difficult to assess from the tracking sheets. TAs were 
asked to record this for the activities of blending, decoding, and word changing, but not for 
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comprehension. TA records were found to often give an indication of appropriate movement 
of pupils in one of two of these areas, but not to give clear evidence of all three. The project 
team therefore had to make a qualitative judgement that, on balance, TAs showed evidence 
of appropriate movement. Their scoring found that 19 out of 49 TAs showed very strong 
evidence, scoring eight or nine out of ten; 21 TAs showed fairly good evidence, scoring 
between six and seven. Only three returned no records at all and two of these were from a 
school that withdrew from the project on the appointment of a new head teacher. 


The tracking sheets also show whether the TAs completed the full 20 weeks of the 
intervention. Records of ten TAs indicate some shortfall, however two of these are missing 
only one day of delivery. Two of the others were from the school that withdrew from the 
project. In the other cases, TAs either job-shared or passed the project over to another TA in 
the school who delivered it to the end point. The indications are, therefore, that the TAs 
delivered the full programme as intended. 


The project team also carried out a full programme of school visits, observing all TAs 
delivering the ICT and non-ICT interventions. These observations used eight criteria to 
assess the sessions: exposure, planning, instructional guidance, opportunities to succeed 
(levels and differentiation), group cohesion, pacing, and efficient use of time and behaviour. 
An overall rating was made of the session. Methods to carry out the assessments are 
described in the footnote.** The project team found the quality of delivery generally high with 
only 10-20% falling below expected standards. These TAs were then given advice and 
guidance on how to improve their delivery of the intervention. This indicates that the project 
was very largely delivered at the expected standard. Differences were found between 
individual teaching assistants rather than between participating schools. 


Adaptations to the programme 


Few variations seem to have been made, partly because the ICT programme was not seen to 
be flexible or allow departures from the schedule. The non-ICT intervention was seen as more 
adaptable, and some TAs altered the balance of activities either towards more blending and 
decoding for lower-ability children, or more comprehension for pupils who were more fluent 
readers. These changes would not appear to affect the fidelity of the programme. They were 
also important in schools’ considerations about whether to continue with the programme after 
the end of the intervention. 


TAs were asked not to share information about their respective intervention with colleagues 
delivering the alternative design. This was difficult for some TAs to follow because the 
activities often took place within schools in full view of staff and pupils. As noted, TAs were 
not aware of the detail of the alternative intervention to their own. It was therefore equally 
difficult to keep other children unaware that there was an intervention which did not involve 
them. However, this was not seen as problematic by TAs because of the young age of the 
children involved. As one TA delivering the ICT intervention explained: 


‘Our children go round and say “we did Abracadabra today” and the 
others are like “what’s that?” and | am like “shush” but it’s been fine, 


cn The project implementation team shot four videos of mock sessions (a different ‘teacher’ each time) in order to 


calculate Inter Rater Reliability (IRR). Project implementation team members then rated each of the four videos 
and calculated the IRR to be 0.71. They identified two sub-scales that had been interpreted differently. They 
consequently changed the sub-scales, re-scored the video sessions, and re-calculated the IRR as 0.85 (95% Cl: 
0.508—0.988). Most of the sessions that they evaluated were observed by two members of the team and an average 
score was used for the final rating. 
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they are children and they are going to talk about something they 
enjoy. The word Abracadabra is quite interesting, but | don’t think 
they know how to explain it. When someone says “what is it?” they 
just say “we go on a computer”. So in all fairness they don’t say 
much.’ 


Little flexibility was offered in delivery and the need for this was understood by TAs. One area 
of flexibility which TAs were concerned to have was in the allocation of pupils to groups 
(within a treatment condition). It was suggested at the training that they might wish to group 
pupils by ability. They were also advised to take account of gender and personalities. At the 
training stage, TAs’ main concern was to ensure that pupils with behavioural problems were 
not grouped together and that groups would function well. Trainers told TAs that they could 
allocate pupils within the two intervention conditions but not move pupils between the two 
interventions. This guidance seems to have been followed. Only six TAs responding to the 
survey said that they had done any rearranging of groups at all. Where they had it was 
because children had left the school or because of the behaviour of individual pupils. 


Outcomes 


TAs expressed different views on which ability groups might benefit most from ABRA. The 
non-ICT intervention was seen to meet the needs of pupils of all ability levels. Pupils of lower 
ability were seen to benefit from its emphasis on blending and coding. Children with very poor 
literacy skills were found to struggle with the level of the text but still to benefit from the 
sessions. Higher-ability children were seen to benefit less from the basic decoding and 
blending activities which were seen as too basic for them, however some TAs felt the 
programme benefited higher-ability children whose technical reading ability is good but who 
have relatively poor comprehension. This was explained with reference to the opportunity for 
discussion in a group setting. That said, a small number of TAs still felt that higher-ability 
pupils required more structured activities within the non-ICT intervention. Some suggested 
that there should be more differentiation for lowest and highest ability pupils. 


The ICT intervention was seen to have something in it for all abilities and to have an 
advantage in this respect over the non-ICT version. Lower-ability pupils were seen to enjoy 
the interactive element of the programme. As one TA explained: 


‘It’s fun. It’s on a laptop so for them it’s exciting. They get to come 
out and do something. They get to click buttons and all sorts and a 
lot of it is actual phonic sounding out which is so brilliant.’ 


However, some TAs thought the pace was too fast for some lower-ability children and too 
slow or repetitive for higher-ability groups. The programme itself was seen as adaptable, but 
TAs wished to follow the guidance and keep to the timings. They also faced the problem of 
making sessions less suitable for one ability group by making it better for another. This was 
less of a problem for the non-ICT intervention since pupils were seen to gain differently from 
the activities and stories. 


The ICT intervention had the advantage of allowing pupils with good ICT skills but weaker 
literacy skills to have a role. This was seen to have the effect of improving the confidence of 
some lower performing pupils. Another perceived advantage of the ICT programme was that 
some pupils viewed it as play, and therefore learned in a more natural, non-resistant way. 
Overall, there was no clear message from the process evaluation about which pupils benefit 
most from either intervention. 


TA views on the impact of ABRA 
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TAs delivering both programmes believed they were having an impact on pupils with most (32 
out of 37) surveyed saying they thought it had a positive or strong positive effect in terms of 
its benefit to pupils. The external evaluation visits shed some light on this finding. Some TAs 
explained this with reference to the frequency of the intervention, at four times a week. TAs 
delivering both the ICT and non-ICT interventions had noticed that pupils were using 
decoding and blending skills in class. However, some TAs felt that, while intense, the 
intervention was too short. As one TA delivering the ICT intervention stated: 


‘If it’s going to make a difference | think it’s going to have to be 
extended a bit.’ 


A number of features of the intervention were seen as promoting learning, in particular the 
group format: TAs felt that this encouraged participation and enabled children to learn from 
each other’s responses. This was reported by TAs in both interventions. Individual pupils 
were seen to have grown in confidence and to be contributing in ways not in evidence in their 
class situation. This was seen to have the potential to impact on their learning. TAs also felt 
they had themselves benefited from being involved in the intervention. In the survey most TAs 
said they felt more confident in delivering small-group work after completing the intervention 
and that they had a better understanding of phonics and children’s language development. 


TAs were asked in both the survey and the visits whether they would continue with the 
programme in the longer term. In the survey 13 out of 32 who responded said they would, 8 
said they would not, and 11 were not sure. TAs interviewed were generally inclined to 
continue but would prefer to do so in a less structured way, for example by using the 
programme less frequently or more flexibly. TAs interviewed during the external evaluation 
visits said they would like to see some small changes to the programme: some believed, for 
example, that the vocabulary presented a barrier to progression for some pupils, particularly 
in the ICT intervention, and felt this would need to be adjusted for pupils to benefit fully. The 
non-ICT intervention was seen as in less need of adaptation, but some TAs felt that timings 
should be adjusted, for example by reducing the length of time spent on each book. 


Formative findings 


The intervention was well implemented and there are no obvious ways in which it could have 
been improved. TAs were well prepared, they clearly understood the purpose, design, and 
delivery plan. They then implemented it as intended. This included aspects such as timing, 
use of materials, and organisation and practical matters. As far as was practically possible, 
the two interventions were delivered separately without contamination in the active schools. 
The project team provided proactive support, and did not wait for problems to be reported. 


Three changes might usefully be made for future implementation of the programme: 


TAs felt that mixed-ability groups worked less well than ability sets in that the latter allowed 
pupils to move onwards at the same pace and not to become bored. Future implementation 
should aim to ensure delivery in ability sets where possible. 


e The vocabulary could be adjusted to remove Canadian terms and made more 
culturally relevant to British children. 

e The delivery of the non-ICT version could be adapted to allow for faster progression 
through the books since this was perceived to be slower than in the ICT version. 


Other changes which could improve implementation are largely beyond the control of the 
programme and need to be addressed by schools themselves. These principally concern 
issues of space—delivery in corridors or in classroom corners is unsatisfactory both for pupils 
and TAs. Schools also need to factor in time for pupils to be taken out of class to attend 
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sessions, and to return. To deliver the ICT programme, TAs require access to reliable PCs or 
lap-tops and a small amount of time for preparation and record-keeping. 


Control group activity 


Data on control group activity was collected both from treatment and control schools through 
a survey carried out by the project implementation team at Coventry University. The survey 
asked lead teachers about the ways in which literacy is taught in their schools. It included 
questions about hours a week spent on literacy instruction, programmes used, and the time 
allocated to activities such as blending, decoding, reading and comprehension. It also asked 
about the use of worksheets, exercise books and other formats, and the use of technology to 
support literacy teaching. Responses to the questionnaire provide an indication of whether 
active and control schools were in any way different in how they deliver literacy teaching. 


Survey responses show few meaningful differences between active and control schools that 
could have affected the intervention results. Both allocated similar amounts of time to literacy 
instruction—around 1% hours a day. Most schools used a combination of whole-class, small- 
group and individual pupil approaches, although the treatment schools said they used more 
whole-class teaching. This finding would seem to discount the possibility that small group 
teaching was more common in treatment schools and that they therefore had a head start in 
this respect. Both treatment and control group schools placed a moderate or great deal of 
emphasis on teaching phonics and had used programmes, in particular ‘Letters and Sounds’, 
‘Read and Write Inc’ and ‘Jolly Phonics’. There were no indications that either group had used 
these to a greater or lesser extent than the other. 


Little difference was found between treatment and control schools in the time they allocated to 
activities such as blending, decoding, reading and comprehension. Control schools reported 
spending slightly less time on decoding, teaching fluency, and reading comprehension than 
treatment schools but more time on ‘reading activities’. They reported spending more time on 
sequencing (putting parts of a story in the correct order), on vocabulary development, and on 
creative literacy activities such as drawing and writing plays. However, the sample sizes are 
small so caution should be exercised in reaching any conclusions from this data. Finally, 
treatment schools were less likely than those in the control group to report that events such 
as the weather and other activities have hindered the teaching of literacy. There were no 
differences in reported use of technology to support literacy teaching, with smartboards 
almost universal, computers used by around two-thirds, and tablets and iPads by a slightly 
smaller number. 


Data collected from TAs in the treatment schools on the scheduling of sessions and what 
lessons were missed by intervention pupils provides additional information. This data 
suggests that the programme did not substitute for core literacy instruction (including phonics 
work) and was used, as intended, as an additional programme, not a substitute for normal 
classroom delivery of literacy. The external process evaluation did not find schools were 
compensating for the programme by delivering additional help to pupils in the control group. 
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5. Conclusion 


Key conclusions 


. The children who received ABRA, or its offline alternative were found to make two and three 
months’ progress in literacy respectively compared to the children who received standard 
provision. This positive result would be unlikely to occur by chance. 


2. For both ABRA and the offline alternative, the impact for children eligible for free school meals 
and children with below average pre-test outcomes was larger than for all pupils. 


. Successful implementation contributed to a well-designed and delivered training programme 
which emphasised fidelity and consistency, enforced by ongoing support from the project team. 


. The process evaluation found that both the ICT and non-ICT interventions may be best delivered 
in groups of similar rather than mixed ability. The process evaluation also suggested minor 
changes to the intervention to make it more culturally relevant to British pupils, and to remove 
some repetition in the non-ICT programme. 


5, Future research will examine whether ABRA or the non-ICT intervention can be successfully 
delivered at scale, and will look at longer-term impacts through assessing Key Stage 1 data from 
this trial. 


Limitations 


The power calculations performed during the design phase suggested recruiting a total of 60 
schools to be able to detect an effect size of 0.20 standard deviations with 80% of power. 
Despite the impressive recruitment efforts of the implementation team at Coventry, these 
numbers could not be met. A total of 48 schools were available for the final analysis. Given 
the data collected from the trial, we calculate that we can detect effects of at least 0.27 
standard deviations with a power of 0.80. However, and as we stated in the protocol, in 
practice we could possibly detect smaller effect sizes. This is because there was a second 
level of randomisation at the pupil level, and the calculations are based purely on a paired 
cluster randomised trial. However, even if we have been able to detect effect sizes smaller 
than 0.20, we might lack power to detect significant effect sizes in subgroup analyses, for 
instance. 


The number of clusters (Schools) available in the final sample should not pose a problem for 
the calculation of clustered standard errors. Dividing the sample to study subgroups where 
the number of clusters is reduced considerably, though, becomes rather problematic and 
results in less than 30 schools and clustered standard errors should be taken with caution. 
This is why we have performed the subgroup analysis by studying interactions instead. 


An additional source of bias in this evaluation results from school attrition. A total of five 
schools dropped out after randomisation. Fortunately, data for all of these schools (except 
one) was collect for post-treatment tests. This means that we have been able to perform an 
intention-to-treat analysis using all, but one, of the initially randomised schools. A further 
problem might result from some pupils not providing consent or leaving school in the middle 
of treatment. Nevertheless, the missing data analysis performed (with a total of 84% of the 
initial sample available for analysis for the main outcome) suggests that both pupil and school 
attrition are not a major source of bias in this evaluation. 


Moreover, for participating schools the process evaluation found that the intervention was 
delivered as planned. Three changes might usefully be made for future implementation of the 
programme: (1) pupils could be allocated to same-ability groups, (2) the vocabulary of the 
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programme could be adjusted to remove Canadian terms and made more culturally relevant 
to British children, and (3) the non-ICT version could allow for faster progression through the 
books. Additionally, some changes which could improve implementation are largely beyond 
the control of the programme and need to be addressed by schools themselves. These 
principally concern: (1) issues of space, with delivery in corridors or in classroom corners 
unsatisfactory both for pupils and TAs, (2) schools need to factor in time for pupils to be taken 
out of class to attend sessions, and to return, and (3) to deliver the ICT programme, TAs 
require access to reliable PCs or lap-tops. 


In terms of the generalizability of the results, we can compare some descriptive statistics in 
our final sample of schools with those of national State-Funded Primary schools (selected 
from the School Level Census 2013/2014). The schools in our sample are bigger on average 
(93 pupils in Key Stage 1 in our sample compared to 74 in State-Funded Primary schools), 
and they have a bigger fraction of FSM-eligible pupils and pupils whose first language is 
known or believed to be other than English. In particular, around 20% of the pupils in our 
sample are FSM-eligible and a similar proportion do not have English as their first language. 
These statistics are almost 16% (for FSM eligibility) and 14% (English is not the first 
language) for pupils in English State-Funded Primary schools. This is in line with the 
recruiting strategy, since an effort was made to encourage schools with disadvantaged 
intakes to participate during the recruitment stage. Therefore we cannot assume our findings 
are generalizable to primary schools generally as this is clearly not a representative sample of 
all schools. 


Interpretation 


The positive findings for the ICT programme are in line with previous studies of the ABRA 
programme. In particular, the ABRA programme has had some support from smaller efficacy 
RCTs (see, for instance, Savage et al., 2009) and a bigger effectiveness trial (Savage et al., 
2013). Both studies found that ABRA leads to improvements in literacy for treatment pupils at 
post-test. This trial, however, differs from the latter in several respects: (1) the size of the trial 
in terms of pupils is doubled, (2) this intervention compares ICT and non-ICT delivery of a 
broadly identical programme—this allows the evaluation of different delivery methods with 
respect to improving results for early literacy, and (3) we have a clean control group—pupils 
in schools that do not receive and do not know about the existence of the web-based 
programme while the intervention is in place. If we have reasons to believe that there might 
be spillover effects from treatment to control individuals within the same school (and we have 
some preliminary suggestive evidence of that) then having a control group in schools that are 
not involved in the intervention gives a much cleaner identification of the treatment effect. 


In the broader literature, there are few rigorous studies about the use of ICT to teach literacy 
that show positive effects. A review by Cassen et al. (2015) states that although there is 
evidence of positive effects of ICT more broadly on English learning, the results are very 
inconsistent and restricted by the rate of ICT use and access in schools. Three relatively 
large-scale studies with a strong methodological design find no effect of teaching with ICT on 
pupil learning (Angrist and Lavy, 2002; Rouse et al., 2004; and Berlinski and Busso, 2014). In 
contrast, the findings in this study will potentially contribute to this broader literature by 
suggesting that an ICT-based intervention has a positive effect, however at this point we 
cannot make strong claims about the impact of technology in this trial. Future research will 
explore alternative data sources to try to understand the difference in magnitude between the 
ICT and non-ICT estimates. 


Even though we find positive effects relating to both interventions, the impact of the non-ICT 
intervention is significantly larger in magnitude than that of the ICT intervention. Several 
factors could be driving these results and further analysis will be performed in the future in 
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order to address the potential mechanisms. For instance, the process evaluation highlighted 
that TAs perceived the non-ICT intervention to be more adaptable to different ability levels. 
Incorporating the data from TA logs and surveys into the statistical analysis could potentially 
help to shed more light on this, and may also help to account for the differential impacts found 
by subgroup, and the preliminary evidence suggesting the existence of positive externalities 
(peer effects) for pupils in the control group of intervention schools. 


With respect to the process evaluation, it provides a number of explanations for the impact 
findings. These principally concern the active engagement of both TAs and pupils with the 
intervention. TAs felt well prepared for the intervention, were supported by the project 
implementation team, and took the responsibility of delivery seriously. TAs felt they benefited 
through being involved in the intervention, improving their skills in small-group tuition and, for 
the ICT programme, their IT skills. They delivered the programmes according to plan and 
moved pupils appropriately through the levels. Other aspects of the delivery were also found 
to be of high quality, maximising the possibility of impact for the programme. 


TAs liked delivering to small groups, they felt it fitted well with their role and liked the 
resources. All of these factors meant they delivered the intervention both diligently and with 
some enthusiasm. This appeared to inspire the pupils who were observed to be actively 
engaged with both the ICT and non-ICT programmes. Pupils enjoyed the small-group format 
led by a TA, the interactive nature of the sessions, the humour and the ‘game’ aspect, and, 
more generally, the appeal of the resources. They appeared to be particularly engaged by the 
ICT programme. However, the attractive features of both interventions are likely to have 
increased their engagement and assisted their learning. 


The process evaluation also found the intervention was delivered as planned and that there 
are no other likely explanations for the higher scores of pupils in the treatment schools. 


Future research and publications 


Future research will examine Key Stage 1 results for treatment and control pupils with the 
goal of determining whether the positive impact found for both interventions persists one year 
after the end of the intervention. Moreover, additional data collected by the implementation 
team in terms of pupil group composition (number of pupils, average ability of pupils, and so 
on), TA surveys, TA logs, and other materials will be incorporated in the analysis with the 
objective of understanding the higher impact of the non-ICT intervention. Moreover, we 
expect to publish academic paper(s) in collaboration with the team at Coventry University. 


Abbreviations: 

IDM: Index of Multiple Deprivation 

PIRA: Progress in Reading Assessment 

DTWRP: Diagnostic Test of Word Reading Processes 
BPVS: British Picture Vocabulary Scales 3 


PSS: Subtest from the Primary Inventory of Phonological Awareness (PIPA): phoneme 
segmentation subtest 


LK: Subtest from the Primary Inventory of Phonological Awareness (PIPA): letter sound 
subtest 


LEST: Letter Sound Test 
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Appendix A 


ICT intervention content and delivery 


A. Content 


The content of the ICT intervention is entirely encapsulated within a web-based 
technology platform, a modular ‘game’-based balanced literacy intervention that 
is fixed in content (new activities cannot and will not be added). The games are 
based around and are linked in terms of content to a series of electronic texts 
(mainly ‘stories’, some non-fiction) suitable to beginner readers. That is, phonic 
fluency and comprehension activities are connected to and draw most content 
from, a series of electronic texts. 


It is important to say that there are no additional ‘extension’ activities beyond 
those presented in the technology (e.g. for writing). One can by viewing the 20- 
week curriculum, the times associated with it, and the software have exactly the 
same experience of the intervention as the participants themselves. 


The term ‘balanced literacy’ is held to mean any programme or curriculum that 
gives, over the course of the intervention, equal attention to ‘word-level’ reading 
skills (including fluency) and broader ‘text-level’ discourse comprehension (and 
fluency) skills. The current programme nevertheless starts with a focus on word- 
level skills in the early weeks and then adds comprehension skills and moves, as 
the programme progresses toward becoming entirely focused on comprehension 
skills as word-level fluency is achieved. 


It is also important to note that ‘word-level’ skills includes skills aimed at 
promoting the phonemic decoding of individual words (the ‘blending train’ 
activity for example) as well as their associated sub-skills (most notably letter- 
sound knowledge and phonemic awareness tasks), but also includes activities 
supporting the direct ‘sight reading’ of words including words that are 
exceptions to phonic rules. These skills and the rationale for including them are 
described in detail in text before Tables 1, 2, and 3 below. 


B. Delivery 


The ICT platform delivers ‘game’-based short literacy activities based around 17 
fiction and non-fiction electronic stories/texts. The intervention described in the 
20-week curriculum document attachment will be delivered through laptop 
computers running the software. The delivery is 4 x 15 minutes x 20 weeks of 
intervention. Unselected samples of ability-grouped year 1 children in small 
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groups of 3-4 will receive the intervention delivered on the computer and 
facilitated by a trained TA precisely following the curriculum and activity 
durations specified in the curriculum document. 


TA training will consist of one full day of university-based training on the 
electronic resource for groups of 8-10 TAs, (9.00-4.00 PM) with 2 breaks plus 
short lunch plus a follow-up half-day session to problem-solve and reiterate 
learning prior to implementation. 


Training day agenda: 


Morning: This session is primarily led by the university facilitator and will detail 
(briefly) the background to, rationale and evidence for, the software resource 
using PowerPoint and questions and group discussion. Then TAs will consider 
elements of literacy (alphabetics, fluency, comprehension), what they look like in 
practice and how they can be taught. This element of the training will be 
sensitive to the fact that not all of the TAs will have prior knowledge and 
understanding of teaching literacy in the UK. TAs will be introduced to the 
software tool directly, be shown basic navigation, and activities and will have a 
short hands-on orientation to the software and discussion before lunch. 


Afternoon: TAs will be provided with laptops with the software on it, will work 
in pairs and will develop a curriculum for a lesson, and then a whole week. They 
will then role-play the lesson to the group who will discuss and co-facilitate the 
activity. Open discussion, and the setting of homework (plan the first 3 weeks) 
will be set prior to a second and a short half-day follow up where TAs will come 
with questions and issues having tried the resource out ‘live’ with children not 
involved with the project. 


The trained TA will have learned by the end of the day how to establish and 
maintain good group work competencies through the use of the STAR principle 
(Sharing, Turn-taking, Active attention, Responding) with a real star (sticker) as 
motivator for these core behaviours, linked to class teachers reward systems. 
The TA will introduce and time monitor activities and will, where appropriate, 
facilitate in the response to very open activities such as story response, 
previewing and predicting, recording successes and establishing whether 
children should progress to the next most difficult activity. 


The intervention as described here and in the 20-week curriculum document 
reflects a cognitive skill-based approach to both word and text-level reading 
competencies but also one where children are exposed to stories that share 
content with all activities (text-linked cognitive strategies). As such the software 
and thus this curriculum was designed to embody much of the most 
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uncontroversial evidence-based practice for early reading as summarized in 
multiple narrative and systematic reviews of the literature (e.g. Ehri, et al.., 2001; 
National Reading Panel, 2000). 


Skills taught in the intervention 


Three forms of broad skills are taught in the intervention: 


A. Alphabetics is the ability to associate sounds with letters and use these sounds to 
create words. The sounds associated with letters are referred to as phonemes (smallest 
units of spoken language) while the written letters associated with these sounds are 
called graphemes. There are 26 graphemes in English but over 40 phonemes. 
Research suggests that a limited number of phonics strategies and letter knowledge is 
essential to acquisition. 


Children begin their path to reading by being able to distinguish between different 
sounds and develop their phonemic awareness. They can begin to associate these 
sounds with print and the correspondence between letters and sounds when they are 
given practice and modeling. 


The ability to manipulate these phonemes either individually (letter by letter) or in 
chunks like rhyming families is a more sophisticated skill in the area of phonics. Here, 
the learner begins to segment words by breaking them apart into units that make it 
easier to read, then blend these units together. Decoding (ability to interpret symbols) 
then begins to take place, as the learner is able to access these strategies to read 
words. Research suggests children who do not have a solid foundation in these 
alphabetic principles are less skilled readers as they progress through school. 


The software resource has 17 different activities specifically aimed at reinforcing the 
alphabetic principles. Each activity has multiple levels so that classrooms with a wide 
range of pupil needs can find an entry point. There is scaffolding built within each 
activity to encourage autonomous use of the tool. 


Many of the alphabetics activities are associated with the leveled stories embedded in 
the software. This helps to build the context for practicing specific vocabulary. Also, 
the activities are appropriate for those pupils who are at the beginning stages of 
alphabetic skills development. These activities would focus predominantly on 
listening skills, auditory discrimination and letter naming. For those pupils who are 
ready for more advanced practice, the activities support word family manipulation, 
decoding games, and blending tasks with text support. Of course, there are fun games 
that provide practice like Letter Bingo and Letter-Sound Search. See Appendices for 
example screenshots of some activities. 


Table 1: Detailed descriptions, leveling and group work for alphabetics activities 
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Activity Description Levels Group 
Name activity 
Letter Given the pronunciation | Level 1 - Upper case | This can be 
Bingo of the letter name, the | letters a group or 
pupil must determine if | Level 2 - Lower case | individual 
the letter is on his/her | letters activity. If 
bingo card. Challenge - Often | played as a 
problematic lower case | group, 
letters (e, d, p, q, g, 1, m, n, | pupils 
u, V) should take 
turns 
finding one 
letter. 
Animated | Pupils will be able to | Level 1: The first page| As above, 
Alphabet | hear the sounds made | presents the 26 letters of | children can 
by certain single and | the alphabet. also support 
clustered letters. These | Level 2: The second page | each other 
letters are then also|has 11 of the most] in this task. 
used in fun alliterative | common consonant and 
sentences (eg. for | vowel clusters for more 
sound /r/ ‘ruth | advanced practice (sh, th, 
rutherford’s red _ robe | ch, or, ck, ai, ee, ea, ow, 00, 
ripped’ (with an | Oa). 
associated amusing 
graphic). 
Letter- Given the pronunciation | The teacher can have 
Sound of a letter sound, the | pupils focus on _ lower, 
Search pupil must identify and | upper, or mixed letters 
find its associated letter | depending on the pupils' 
‘hidden’ (entirely | needs 
visible but placed in|Level 1 - 10 letters 
details) in a picture. camouflaged in an easy 
background; i-e., s, p, m, t, 
c,d,n,a,hr 
Level 2 - 14 letters 
camouflaged in a medium 
background; i.e., g, l, 0, k, 
u, b, e, f, 1, z, d, h, p, r 
Level 3 - 14 letters 
camouflaged in a _ hard 
background; i.e., w, j, V, y, 
q, x, g,i,b,la,t,e,h 
Same The pupil will be| All letter sounds are | This activity 
Phoneme | required to distinguish | presented in this activity. | can be used 
between phonemes by pairs 
(letter sounds) that are (alternating 
the same or different. after doing a 
pair of 


words) and 
as a whole 
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pupils will identify the 
word it makes; e.g., /r/ 


/o/ /d/ = "rod". 


NOTE: TA / peers must 
judge accuracy’- of 
articulation. 


If child struggles with 
task as a_ purely 
phonological one then a 
‘help’ button provides 
the letters to go with 
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vowel sounds. (e.g. a-t, ea- 


t) 

Level 2: 2-phoneme 
words with clusters, 
short, long, and_e r- 


controlled vowel sounds. 
(e.g. (e.g. ar.-t, t-ar) 
Level 3: 3-phoneme 
words with short vowel 
sounds (e.g. c-a-t-) 


Level 4: 3-phoneme 
words with clusters, 
short, long, and_=e r- 


class 
activity 
Auditory | Given a_- phonemic| Level 1: 2-phoneme | Pupils can 
Blending | breakdown of a word,| words with clusters, | do this 
the pupil will identify a) short, long, and_ r-| activity in 
word; i.e. given /c/ /a/ | controlled vowel sounds | pairs and 
/t/ child chooses "cat" | (e.g. a-t, ar-t). take turns 
from among 4 picture | Level 2: 3-phoneme | (one’ word 
choices words with short vowel | per turn) to 
NOTE: if child get task | sounds. (e.g. c-a-t) tell the 
wrong number’ of)|Level 3: 3-phoneme|word and 
choices reduces from 4 words with _ clusters, | find its 
to 3 (and as needed to | short, long, and_ r-| matching 
2) as support. controlled vowel sounds | picture 
(e.g. c-ar-t) 
Level 4: 4-phoneme 
words with blends mainly 
at the beginning, short, 
long, and_ r-controlled 
vowel sounds. (g-r-ai-n) 
Level 5: 4-phoneme 
words with blends mainly 
at the end, short, long, and 
r-controlled vowel 
sounds. (m=i-!-k) 
Level 6: 5-phoneme 
words with blends at the 
beginning and end, short, 
long, and_ r-controlled 
vowel sounds (e.g. p-l-a-n- 
t). 
Level 7: 5-phoneme 
challenge words (e.g. t-ar- 
g—e-t) 
Blending | Given the phonemic| Level 1: 2-phoneme|As well as 
Train breakdown of a word, | words with short and long | group work, 


pupils can 
do this 
activity in 
pairs. They 
take turns 
(one word 
per turn) 
listening to 
the 

breakdown 
of a word 
then putting 
the sounds 


the sounds as_ they | controlled vowel sounds | together to 
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word read. 


The help button in this 
case gives the sounds of 
the letters sequentially 
allowing the child just 
to focus on the blending 
and the selection of the 
correct picture to go 
with the word. 


controlled vowel sounds. 

Level 4: 4-phoneme 
words with blends mainly 
at the beginning, short, 


long, and_ r-controlled 
vowel sounds. 
Level 5: 4-phoneme 


words with blends mainly 


at the end, short, long, and 
r-controlled vowel 
sounds. 

Level 6: 5-phoneme 


words with blends at the 
beginning and end, short, 


long, and_ r-controlled 
vowel sounds. 
Level 7: 5-phoneme 


challenge words. 


blend them. (e.g. c-ar-t) tell the 
Level 5: 4-phoneme | word _ they 
words with blends mainly | make. 
at the beginning, short, | Equally _ all 
long, and_ r-controlled | peers or 
vowel sounds. (e.g. gr-ai- | pairs can 
n) judge 
Level 6: 4-phoneme | whether 
words with blends mainly | word 
at the end, short, long, and | supplied by 
r-controlled vowel | other child 
sounds. (e.g. m-i-l-k) is correct or 
Level 7:  5-phoneme | not. 
words with blends at the 
beginning and end, short, 
long, and_ r-controlled 
vowel sounds (e.g. p-l-a-n- 
t). 
Level 8: 5-phoneme 
challenge words (e.g. t-ar- 
g-e-t). 
Basic Given the visual | Level 1: 2-phoneme | This activity 
Decoding | representation of a|words~ with clusters, | combines 
word, the pupil must) short, long, and_ r-| the 
say the sounds of the | controlled vowel sounds. | sounding 
letters then blend them | Level 2: 3-phoneme | out of 
together to read the | words with short vowel | letters and 
word. They must then | sounds blending 
find the correct picture | Level 3: 3-phoneme | them to 
(among three incorrect | words with clusters, | make 
pictures) to match the | short, long, and_ r-| words. The 


teacher or 
other pupils 
can assist 
with 
scaffolding 
or picture 
matching 
when 
necessary. 
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individual letters in a 
word to form a new 
word. First a_ child 
identifies the letter that 
needs to be changed 
(e.g. when asked to 
change ‘bow’ to ‘cow’ 


Note: help is provided 
by computer if wrong 
letter is chosen (first, 
sounds given, or choice 
reduced in second tier 
of support, finally task 
is modeled if child still 
not successful) 
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the first letter is 
manipulated (e.g. ‘b’ ‘and 
‘r’ in bat-rat) 

Level 2 - CVC words. All 
letters are manipulated 
(e.g bat-rat bat-bag, bat- 
bit) 

Level 3 - CVC words with 
long vowel sounds. Only 
the first letter is 
manipulated (e.g. dine- 
line) 

Level 4 - CVC words with 
long vowel sounds. All 
letters are manipulated 
(e.g. dine-line, dine-dive, 


can be done 
individually 
or in pairs. 
If a child 
makes a 
mistake, the 
teacher or a 
peer can 
encourage 
the pupil to 
pay 
attention to 
the required 
sound. 


dine-dune). 


B. Comprehension 


Comprehension is the cognitive process in which the reader interacts with a text in an 
attempt to understand what is taking place in the writing. Reading comprehension is a 
culmination of all the pupils’ reading skills that have been gained. It involves good 
oral comprehension, vocabulary and decoding skills. (Depending on the grade level of 
the pupils, the importance of each of these prerequisite skills varies.) Comprehension 
is also affected by children’s prior knowledge and personal experience that can help 
them relate to varied texts. 


Reading for meaning is the ultimate goal of learning how to read. This is the element 
that takes children from being good decoders to being good readers and literate 
beings. 


Research suggests that there are several strategies that have a positive influence on 
children’s ability to understand what they read. The National Reading Panel (2000) 
suggests practicing skills like asking and generating critical questions, using story 
maps, and monitoring comprehension through cooperative learning opportunities. 


There is also evidence that modeling appropriate strategies such as using context clues 
and allowing children to gain practice through providing them with multiple 
opportunities to interact with texts. Pupils should have time to read individually, with 
peers and with adults. Content here thus reflect these forms of evidence-based 
strategy. 


This technology has focused on developing several specific skills that contribute to 


overall comprehension. These activities are leveled and range from simpler tasks like 
placing a well-known story in order (beginning, middle, and end) to summarizing an 
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entire text. Other activities allow pupils to respond to a question prompt encouraging 
pupils to think critically about the text and respond appropriately given the context of 
the story. In addition, the vocabulary activity helps pupils build a bank of words that 
they may be able to decode but do not know the meaning. See Appendices for 
example screenshots of some activities. 


Table 2: Detailed descriptions, leveling and group work for comprehension 


activities 


Activity Name 


Description 


Levels 


Group activity 


Prediction 


Based on 
information from 
the story, the 
pupils will predict 
future events. 


Comprehension 
Monitoring 


After reading a 
story, the pupil 
will identify 
words that do not 
make sense. 
(There is one 
incorrect word on 


each page.) 


There are no 
‘levels’ as such to 
this activity. 
Children predict 
for the story they 
are working with 
in 20-week 
curriculum. These 
stories are 
however in order 
of length in the 
curriculum so 
comprehension 
tasks increase in 
difficulty 
throughout for 
prediction/ 
summarizing 
activities etc. The 
story order is: ‘T 
can move like a....’ 
(an imitation 
activity), 
Beansprouts (non- 
fiction), The Dove 
and the Ant, The 
Frog and the Well, 
Henny Penny, the 
Little Red Hen (all 
fairy tales), 
Waterfalls (a 
longer folk story) 


This activity is 
first done as an 
oral project. 
Pupils can also 
read the story as 
a group using the 
tracking function 
or have the page 
read by _ the 
computer using 
the audio icon. If 
done orally, 
encourage pupils 
to discuss their 
ideas with peers. 


Pupils can work 
as a group 
reading the story 
and looking for 
the nonsense 
words and take 
turns to be in 
control of the 
mouse and 
computer. If a 
word is not 
known, they can 
click on it to get 
help. 


Sequencing 


After reading a 
story, the pupil 
will place story 
events in their 


correct order. 


Level 1: 3 story 
events (summary 
of story pages), 
displayed in 
random order. 

Level 2: 5 story 
events (summary 


Pupils can work 
as a group 
(reading the story 
events and 
discussing the 
appropriate 

order) and_ take 
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of story pages),|turns to be in 
displayed in| control of the 
random order. computer. 

Summarizing The pupil will | Story related | This activity is 
answer story | questions are | best done in pairs 
related questions | asked after certain | or in small 
to help him/her | pages in the story. | groups. Pupils 
summarize the read the story 
story at the end. (using the 

tracking function) 
or have the pages 
read by _ the 
computer (audio 
feature). TA 
encourages 

discussion among 
the pupils. 

Vocabulary Given the | Some words are | This feature can 
definition of aj| selected for each | be used to 
word, the pupil | story. preview 
must determine vocabulary words 
which of two from a particular 
sentences uses the story before 
word reading it or to 
appropriately. work with words 

from a story after 
it was read. 

Story Response Pupils are given | Six questions are | This activity 
open-ended presented in  aj| should be done in 
questions about | set—three of | small groups. 
the story that they | which are general | After pupils have 
will discuss with | while the | discussed a 
one another (e.g. | remaining three | question, they 
did you ever try to | are story-specific. | take turns scoring 
help someone like goals. 
the little red hen 
did?) 

Story Elements The pupils must | Pupils must select 
answer questions | the answer to six 
about events that | questions about 
took place in a/| the story. Because 
story (e.g. how did | there are audio 
the story end?) prompts to assist, 

this activity can be 
done by all pupils 
once they are 
familiar with the 
story. 
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C. Reading (Fluency) 


Fluency describes the level of automaticity that children have when reading a text. It 
is when learners are able to decode with little or no effort helping them to concentrate 
on the content of what they are reading as opposed to figuring out words and sounds. 
The sub-skills under the umbrella term fluency add to this skill correlating to 
comprehension. Pupils who read with correct pacing and expression with few or no 
mistakes are better able to focus on the meaning of the text. 


Research suggests that pupils who are unable to acquire the skill of automatically 
reading words will struggle with attending to meaning. In addition, children who 
struggle with reading accurately and at an appropriate pace tend to struggle with 
motivational problems (Reading Rockets). The reading load increases as pupils 
progress through the grade levels making lack of fluency a major obstacle in school 
success. 


The technology has a range of activities that allows for practice in reading fluency. 
The activities cover all of the sub-skill areas, including expression and speed. 
Through fun games, pupils can test their pace against cartoon characters. The 
technology has models of reading built into the story-related activities so pupils can 
monitor where they can improve. In addition, the programme models how to decode 
words within the context of the story so they are receiving additional and consistent 
practice at figuring out words. The more leamers are able to practice a specific skill, 
the better the chance it has of becoming automatic. See Appendices for example 
screenshots of some activities. 


Table 3: Detailed descriptions, leveling and group work for Fluency activities 


Activity Name Description Levels Group activity 


High Frequency 


Given a list of high 


Level 1: A set of 


Pupils can do this 


story, the pupil 
will be able to 


appropriate for all 
stories and levels. 


words frequency words, | four words are | activity in groups. 
pupils will have to | randomly TA reminds pupils 
read them _ out | presented 4 times | that speed in their 
loud within = a | with gradually | word recognition 
certain time | less time to read | ability is an 
period before | them important 
another word | Level 2: A set of | element in their 
pops up. seven words are | becoming good 
randomly readers. 
presented 4 times 
with gradually 
less time to read 
them 
Tracking While reading the | This activity is | Pupils can take 


turns to read the 
story and be in 
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skipping words. 
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control of the 
computer (peers 
can monitor 
accuracy). 


Accuracy 


After listening to 
the computer read 
a passage, the 
pupil must read 
the same _ page 
accurately. 


This activity is 
appropriate for all 
stories and levels. 


This activity 
should be done 
individually (but 
peers can monitor 
and take turns 
reading it with 
least capable 
going last and 
benefiting from 
repeated 
observation of 
task). As the 
computer reads, 
the pupil follows. 
When s/he reads, 
s/he can click on a 
word if help is 
needed. At the 
end of each 
passage, the 
words clicked will 
be reviewed in a 
game-like feature 
to help the pupil 
become more 
familiar with 
them. 


Expression 


The computer reads 
a text using 
different 
expressions and the 
pupil must decide if 
the passage was 
read appropriately. 
The pupil must then 
read the same text 
practicing correct 
use of expression. 


This activity is 
appropriate for all 
reading levels. 


This activity can 
be carries out in 
turns by _ pupils 
and decision 
made collectively, 
or completed in 
pairs. 


Reading Practice 


This activity 
exposes pupils to a 
new story, which 
will allow for more 
reading 

opportunities and 
practice. This 


Suitable for skilled 
readers at end of 
intervention. 
NOTE pupil who 
can read new 
passages 
independently 


Repeated reading 
is shown to be an 
effective strategy 
in the 
development of 
Fluency and other 
reading skills. We 
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activity can be done | and who have | suggest that you 
in pairs or in small | completed all | have pupils 
groups and | other activities | reading passages 
individually. may be eligible to | orally = multiple 
‘graduate’ (i. e.| times while 
discontinue) the | receiving 
programme and | guidance or 
return to full-time | feedback from 
classroom peers, parents, or 
teaching. teachers. 


Progression in activities 


There is a very closely planned progression in both word-level and text- 
level activities from the less to the more complicated (with some elements of 
progression also in fluency activities). The aim in this is to allow all children to 
have appropriately challenging material throughout the intervention for their 
developmental level. These progressions are detailed for all lessons in a 20-week 
lesson plan (see ‘Detailed 20-week Lesson Plan attachment’). As shown there, of 
each 3 elements of literacy (alphabetics, fluency, and comprehension) are carried 
out as streams that run at the same time through the 20-week intervention. 
There is a general overarching progression in the sense that at the star of the 
intervention there is a greater focus on children’s word reading and associated 
alphabetic skills. There is a form of gradual ‘hand-over’ towards fluency and 
comprehension activities (that is, more time per 15 -minute session is devoted 
to these two activities later in the intervention as children’s word reading 
progresses). 


Beyond this the progression within the three streams is governed by a 
general rule. The general rule is that children should on any given task be 
working at around 90%+ accuracy (this figure is drawn from quite conventional 
curriculum based assessment routines). After 3 successes at this level on any given 
level a child (or group) moves on to the next level up in the ladder of difficulty. 
Typically this is monitored by the TA and it means that children can remain at a 
level until competency is evident. It can also mean on occasion that children (or 
groups) move backwards though activities if they have not shown sufficient 
competence at a given level (less than 80% accuracy at a given level). 
Progressions within streams are described below: 


A) Alphabetics/Phonics progressions: This progression principle is easily 
mapped in phonic activities (that is, letter-sound knowledge and 
phonological awareness activities) described in Table 1. For 2 of the 3 
letter-sound activities (letter-sound bingo and animated alphabet) the 
progression is as follows: 
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Level 1 90% success rate 3 times in a row: move to next level 
Level 2 90% success rate 3 times in a row: consider discontinue 
task 


For letter-sound search the progression is as follows: 


Level 1 90% success rate 3 times in a row: move to next level 
Level 2 90% success rate 3 times in a row: move to next level 
Level 3: 90% success rate 3 times in a row: consider discontinue 
task 


For the majority of the phonological and decoding activities (namely the 
auditory blending, blending train, and basic decoding the progression is as 
follows: 


Level 1: 2-phoneme words with clusters, short, long, and r-controlled vowel 
sounds. 

Level 2: 3-phoneme words with short vowel sounds 

Level 3: 3-phoneme words with clusters, short, long, and r-controlled vowel 
sounds. 

Level 4: 4-phoneme words with blends mainly at the beginning, short, long, and 
r-controlled vowel sounds. 

Level 5: 4-phoneme words with blends mainly at the end, short, long, and r- 
controlled vowel sounds. 

Level 6: 5-phoneme words with blends at the beginning and end, short, long, and 
r-controlled vowel sounds. 

Level 7: 5-phoneme challenge words. Level 1 


In each case, 90% success rate 3 times in a row: move to next level, 
(consider) discontinuation at level 7. 


For basic decoding there is a 4-level version of the same principle with the 
same 90% success discontinue rule: 


Level 1 - CVC words. Only the first letter is manipulated (e.g. ‘b’ ‘and ‘r’ in 
bat-rat) 

Level 2 - CVC words. All letters are manipulated (e.g bat-rat bat-bag, bat-bit) 
Level 3 - CVC words with long vowel sounds. Only the first letter is 
manipulated (e.g. dine-/ine) 

Level 4 - CVC words with long vowel sounds. All letters are manipulated (e.g. 
dine-line, dine-dive, dine-dune). (Possible) discontinuation here. 


Please note that as the web-based tool was designed to be entirely modular these 
common structural features of programmed learning are not built into the 
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system but have to be superimposed through the curriculum and the adult 
supervising the sessions. 


The precise details of these progressions are based on the sum of much research 
in the 1980s and 1990s on development of skills such as phonological awareness 
and word reading accuracy (full citations can be given for reviews of this work). 


The progression for word-level skills is essentially built on a widely-accepted 
developmental model of the combined causal influence of letter knowledge and 
‘levels of phonological awareness’ from simple phoneme blending tasks to 
complex phoneme segmentation tasks and their joint deployment in phonic tasks 
of graduated increased difficulty. Adjacent to these are sight word reading and 
text exposure activities that help to consolidate these skills in reading. These 
latter tasks at the higher end include reading text with appropriate speed and 
judging the correct prosody and emotional tone of a passage (and as such are 
classified as fluency activities here in this document). 


B) Comprehension progressions 


With the exception of sequencing which has two levels and which follows the 
same 90% progression rule articulated above for alphabetics), most 
comprehension activities are more open-ended and not all of them readily lends 
themselves to simple ordered progression rules, (though the accuracy on 3 
successive occasions is still a useful heuristic where applicable, and the 90% rule 
can apply to Vocabulary, Story Elements and (to an extent) to Story Response). 


More generally then, the progression for text-level skills as described in detail in 
the 20-week intervention) is essentially in the development of individual word 
knowledge (vocabulary) and easier ‘receptive’ tasks (no spoken response 
needed) of story ordering (e.g. 3 element stories) through after success at the 
first level, to longer and more challenging 5-element stories and comprehension 
monitoring activities to more difficult ‘expressive’ (verbal response) tasks of 
answering specific questions about a text and finally to key skills of predicting 
and summarizing parts of texts, and responding as a reader to texts. 
Comprehension tasks thus also graduate children to tasks of increased difficulty 
as they succeed. 


Some task ordering is determined by logic: Prediction before a text (or part of a 
text) is read, Summarizing, and story response after reading text or text element. 
Some activities (e.g. Vocabulary) can occur in both places, and some are best- 
used in real time during reading (like Comprehension Monitoring), so are 
ordered around each of the texts thus. 
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C) Fluency Progressions 


The progression for fluency tasks is similar to that for the comprehension tasks 
in that they generally do not lend themselves to progression rules in the same 
way that phonic tasks in section A do. 


Additional considerations 


In all cases, for the precise progression of activities see the 20-week curriculum 
and the associated descriptions of each activity. Note that in all case children 
experience a rewarding (within technology) activity that follows the one they 
have completed (e.g. shooting a hockey puck at a goal after answering a 
comprehension question well), and children are often given some motivating 
choice of (selected) tasks at the end of a week of work. 


It is important that the tool is used flexibly, to meet pupil needs, progression and 
discontinuation rules go some way towards this end of making sure the tool is 
not one size fits all and targets optimal levels for children who will arrive at 
different levels and progress at different speeds. Ability grouping should allow 
progressions to be similar across groups though an element of judgment of 
where the majority are operating will be needed. The progressions in the 
curriculum document assume a low-level of skills but it may clearly emerge that 
skills have been mastered in a given group (e.g. basic blending, all letter-sounds) 
and some testing of appropriate level needed to make sure pupils are not held 
up. Some of the information in this document can be given as a ‘fact sheet’ to TAs 
to aid comprehension. 


Record Keeping 


TAs will be encouraged to keep an activity log, detailing the activity of each 
session (see Appendices). The ICT intervention has built in to it a record keeping 
facility for both teachers and researchers. The teaching assistants can login to a 
teacher area of the software and access a teacher report that keeps track of the 
activities, levels, and performance data. There is a more detailed level of tracking 
data that can be accessed by researchers that tracks pupils, activities, levels, 
performance, and time spent on each activity. 
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Appendices 


Example characters in the 


resource 


Julie, the main character 


These characters animate activities for children and provide a linking theme to 
the resource that children can engage with. 
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Example screenshots from ICT intervention activities 


Once upon a time, Little Red 
Hen lived on a farm with a 
dog, a pig and a cow. 


An example of a page that can be read by (or, by using the bird icon, to) child: 
The Little Red Hen story 
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An example of the phonic assistance given in stories if one clicks on a word 
(phoneme-graphemes are given for decodable ‘regular’ words, letter names for 


exception words). 


Blending activities 


An example of the leveling of activities in phonics, this one is for the Blending 
Train activity. 
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An example of the auditory blending activity for word ‘wet’ where bubbles are 
for the phonemes and the 4 choice answer pictures below are indicated. Children 
can click to hear the phonemes ‘w’-‘e’-‘t’ as often as they need. 


Word changing Activity 


Here, an example page from the Word Changing activity, here a child is 
supported in turning the word ’make’ into ‘made’ by fishing for the correct letter. 


Comprehension Activity 
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This is the 3-item story ordering activity for Comprehension. Children click on 
the story sections 1-3 to hear them and then place them into the appropriate 
order. 
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ICT-led TA Literacy Lesson 


Activity Log 
TA’s Name: Year Level: 
Date and time: Activity: 
Number of children participating: ANY ABSENT CHILDREN? 
Location of lesson: Time taken to introduce the 
activity: 


Please check the specific activities you undertook during this lesson. 


Word Level 
Activities 


Blending 


Decoding 


Letter-sounds 


Segmenting 


Sight words 


Specific activity name 


Additional comments on the above: 
Performance Probes 
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Level and 
(group) success 
rate (90%+?) 


Text Level Activities 


Story 


ordering 


Tracking 
Summarizing 


Comprehension 
Monitoring 
Story 


response 


Reading 


with 
expression 
Prediction 
Sequencing 
Vocabulary 
development 


Story 


elements 


Success level / rate?: 
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How do you feel the technology lesson progressed today? (Were all the children 
engaged? What aspects were you particularly pleased / displeased with?), Can 
children progress? (what evidence is there for this?) 


Did you experience any technical difficulties? How was this handled? How much 
time was taken to resolve the difficulties? 


Did the session go as prepared? What is your comfort level with the programme 
and activities that were being used during this session? 


Please share any other observations you may have had regarding teaching and 
the children in regards using the technology i.e. class management, organization, 
teaching style, unforeseen interruptions, ability level differentiation. 


Education Endowment Foundation 77 


Education Endowment Foundation 


ABRA: Online Reading Support 


78 


ABRA: Online Reading Support 


Appendix B 


Non-IC Tintervention content and delivery 
Content 


The non-ICT intervention is defined by a 20 week programme of lesson 
plans based on pencil and paper activities, with additional resources 
such as magnetic letters and cards. The activities are linked to a series 
of texts. All resources and lesson plans are provided in a bespoke pack 
supplied to each TA ata training event prior to the intervention. 


The intervention is a balanced one in that it encompasses both low 
level decoding activities and higher text level comprehension 
activities. 

The content of the lessons are divided into 3 categories: 


Decoding tasks (D) - to develop the ability to interpret 
symbols such as letters. 


Comprehension tasks (C) - to develop skills in interacting with the 
text to gain meaning from the story. 


Reading tasks (R) - to develop levels of fluency when 
reading text. 


Activities are designed to be suitable for children with a range of 
abilities by dividing the activities into different levels, and pupils 
progress through the levels in a systematic manner. 


Delivery 


The lessons plans are designed to be delivered to groups of 4 pupils 
with each session lasting 15 minutes. Each lesson is carefully divided 
into timed sections and therefore each teaching element is designed 
to be achievable in the time give. Being able to remain within the 
allotted amounts of time is important as Teaching Assistants will have a 
number of groups to work through in an aftemoon. 
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The lessons plans are designed to be delivered by Teaching Assistants 
who will have received a full days training on the content and delivery 
of the maternals. The Teaching Assistants will continue to be supported 
by just-in-time support by members of the research team should they 
need assistance during the intervention penod. 


Training 


Each of the Teaching Assistants will be provided with a resource pack 
at the October training session which will contain all the paper based 
matenals they require for the 20 weeks programme. This includes daily 
lessons plans for each of the 15 minute sessions and copies of the 
books they will share with the pupils. 

Training will consist of a one day workshop to go through the 
intervention and programme. Plus a half day follow-up training after 
the TAs have had a chance to practice some sessions on non- 
participating children. Training will be sensitive to the fact that not all 
TAs will have received pnor training in teaching literacy in the UK. 


It is acknowledged that the Teaching Assistants who will be delivering 
this intervention will have differing levels of experience in teaching 
literacy skills that are required for the activities. Therefore a Key Facts 
sheet will be provided (see Appendix 1). The information in this sheet 
will also be covered in the training session prior to the intervention 
beginning. 


An important part of the training which will be provided for Teaching 
Assistants before they begin to deliver the programme will focus on 
ensuring they feel confident to start pupils on the correct decoding 
levels, as well as how to progress through the levels. 


Skills taught in the intervention 


The content of the lessons are divided into 3 categones, and each 
category hasa numberof activities in the lessons plans related to it. 


Decoding 
The decoding related tasks in the programme acknowledge that pupils 


will be entering with differing levels of skills (for example some pupils 
may begin the programme having consolidated all the initial letter 
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sounds whereas others may still have gaps in their letter sound 
knowledge). These lesson plans are designed to provide flexibility for 
the Teaching Assistants to start pupils on decoding activities which are 
appropnate for their current level. This will ensure that pupils do not 
become bored completing work they have already consolidated or 
frustrated through being offered activities they cannot complete. 


In terms of the Alphabet activities (which introduce different phonemes 
and are used ona daily bass) if itis agreed that the pupils in the group 
have already consolidated these skills the amount of time allotted to 
this activity will be reduced to 1 minute to act asa quick and familiar 
starter activity foreach session. As pupils progress through the Alphabet 
activities it may be necessary for the Teaching Assistant to increase the 
amount of time again if pupils begin to encounter phonemes they 
have not consolidated. If time is reduced on this activity it is added to 
the either the Comprehension or Reading activity section (this is noted 
on the lesson plans). 
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Activities Description Resources 
Letter Bingo Given the pronunciation of a | Photocopiable bingo boards 
letter name, the pupils must | and letters are provided in TA 
determine if they have the | resource pack. 
letteron their bingo card. 
Pupils must identify  initial/ | Alphabet letter flashcards are 
Alphabet middle orfinalsoundsin words. | provided in the TA resource 
pack. 
Given the phonemic 
breakdown, the pupils will] For both Blending = and 
Blending identify a word. Decoding activities TAs are 
provided with a range of 
suggested suitable activities 
Given the visual representation | which they can chose from 
of a word, the pupil must say | depending on the needs of 
Decoding the sounds of its letters, then | the group they are working 
blend them together to read | with and the _ resources 
the word. available in the school they 
are working in. 
The pupils will distinguish | No resourcesrequired. 
between letter sounds that are 
the same and different. 
Same Sounds 
Given the pronunciation of a 
letter sound, the pupils must | Photocopiable activity sheets 
identify and find its associated | are provided in the TA resource 
letter. pack. 
Letter-Sound 
Search Given a high frequency word 


High Frequency 
Words 


Word Changing 


the pupils will read it correctly. 


Pupils change individual letters 
in a word to forma new word. 


Photocopiable high frequency 
word flashcards are provided 
in the TA resource pack. 


The TA can use either a 
whiteboard or magnetic letters 
to create word changing 
words. A list of suitable words is 
given in the TA resource pack. 


The programme is designed to ensure that pupils progress to the next 
level of activities once they have (as a group) achieved an 80-90% 
success (repeated 2 to 3 timesina row). 


In terms of the Blending and Decoding activities pupils work through 7 
levels: 
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LEVEL ACTIVITY 

1 2-phoneme words with clusters, short, long, and Fr 
controlled vowel sounds 

2 3-phoneme words with short vowel sounds 

3 3-phoneme words with clusters, short, long, and Fr 
controlled vowel sounds 

4 4-phoneme words with blends mainly at the beginning, 
short, long, and rcontrolled vowel sounds 

5 4-phoneme words with blends mainly at the end, short, 
long, and controlled vowel sounds 

6 5-phoneme words with blends at the beginning and end, 
short, long, and controlled vowel sounds 

Fi 5-phoneme challenge words 


Table 2: Blending and Decoding Levels 


As with the Alphabet activities Teaching Assistants will be supported 
through the initial training to identify a suitable starting point for pupils. 
They will then work through the different levels recording the results of 
each session on the Tracking Sheet (see Appendix 2) to monitor when 
pupils have achieved the group success rate and need to move up a 
level. No specific blending activities are given in each lesson plan 
instead a list of appropnate activities are provided (see Appendix 3). 
This will ensure that Teaching Assistants are able to select activities 
which will be motivating to the groups they are teaching and which 
the school have appropnate resources for. 


In terms of the Word Changing activities the pupils work through 4 
different levels: 


LEVEL ACTIVITY 
CVC words. Only the first letter is manipulated. 


CVC words. Allletters are manipulated. 


WIN |R 


CVC words with long vowel sounds. Only the first letter is 
manipulated. 


4 CVC words with long vowel sounds. All letters are 
manipulated. 


Table 3: Word Changing Levels 


It is intended that pupils will all begin at Level 1 with the Teaching 
Assistant moving them through the levels when they have achieved 
the 80-90% success rate (repeated 2 to 3 times in a row). There are 
example words given for each Word Changing level which Teaching 
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Assistants can use and then adapt if necessary (see Appendix 4) and 
also a Work Changing Tracking table (see Appendix 5). 


Comprehension 


A range of comprehension activities are contained within the lesson 
plans. In order to support Teaching Assistants in the comprehension 
activities prompts sheets (Known as Top Tips sheets) will be used (similar 
sheets will also be provided for the Reading activities). These will be 
same across both the Teaching Assistant led and the computer based 
programme conditions (for an example Top Tips Sheet see Appendix 


6). 


Vocabulary 


Sequencing 


Monitoring 


Prediction 


Story 
Response 


The pupils will explore 
the meaning of single 
words. 


After reading a story 
the pupils will place 
story events in their 
corect order. 


After reading a story 
the pupils will identify 
words that do and do 
not make sense. 


Based on information 
from the story, the 
pupils will predict 
future events. 


Pupils are given open- 
ended questions 


Single pages from each story 
with the target words 
highlighted are provided in the 
TA resource pack. 


A senes of either 3, 5 or 8 story 
cards are provided in the TA 
resource pack for pupils to 
order. 


Pupils are either presented with 
a picture and sentence from 
the story with a word missing 
which they must identify from 
the options given or presented 
with questions about the story 
which must be discussed as a 
group and answered. These are 
provided in the TA resource 
pack. 


The TA will read through a new 
story with the pupils asking the 
prediction questions provided in 
the TA resource pack. 


A list of question to ask pupils is 
provided in the TA resource 
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about the story that| pack. 
they will discuss with 
one another. 


Story The pupils must | A list of questions to ask pupils is 
Elements answer questions| provided in the TA resource 
about events in the | pack. 
story. 


Most of the comprehension activities are not explicitly structurally 
levelled like the decoding activities, with the exception of the 
Sequencing activity. 

However, children still make progress through the comprehension 
activities through the incremental complexity of the sequence of 
activities as presented through the 20 week plan (available on 
request), and the complexity of the stories. Forexample, early activities 
are limited to single word level semantic work (Vocabulary) whereas 
later activities such as predicting future events in texts (Prediction) are 
more challenging. 


Reading 


Throughout the 20 week programme a range of books are used to 
support the reading activities (see attached file of books). The titles of 
the booksare as follows: 


Weeks 1& 2-1Can Move Like A.... 
Weeks 3, 4 & 5- How a Bean Sprouts 
Weeks 6, 7 & 8- The Dove and the Ant 
Weeks 9, 10 & 11 - The Frogsand the Well 
Weeks 12, 13 & 14 - Henny-Penny 

Weeks 15, 16 & 17 - The Little Red Hen 
Weeks 18, 19 & 20 - Waterfall 


Activities Description Resources 
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The pupils will read with 
one-to-one 
comespondence and 
without skipping words 


After listening to a passage 
being read, the pupils will 
read the same accurately. 


The pupils will listen to a 
passage being read and 
decide whether it is being 


The pupils with read 
from the stories 
provided in the TA 
resource pack. A list 
of tracking tips is 
provided forthe TAs. 


The pupils will use the 
stones provided in the 
TA resource pack. 


The TA will read from 
the stones provided in 
the TA resource pack. 


read with good or bad 
expression. The pupils will 
then read the passage. 
Speed The pupils will read the text | The pupils will use the 
atanappropnate speed. | stones provided in the 
TA resource pack. 


Each pupil in the group will require access to a paper copy of the 
book. A range of reading activities are provided throughout the 20 
week programme. As with the Reading activities Teaching assistant will 
be provided with a ‘Top Tips’ sheet for each activity and these sheets 
will be the same across both the Teaching Assistant led and computer 
based programme conditions. 


Additional considerations 


Within the lessons plans from week 12 onwards Teaching Assistants will 
be given more flexibility in deciding which activities to present to pupils 
depending on levels already achieved. Teaching Assistants can 
choose between Decoding, Word Changing or Tracking activities. 
Training on how to choose the comrect activities will be given at the 
training event. In addition to this, support will also be provided around 
week 10 to staff by a Senior Research Assistant either through fac e-to- 


Education Endowment Foundation 


86 


ABRA: Online Reading Support 


face meetings in school settings or through a newsletter to ensure they 
feel confident in choosing the corect activities. 


It is acknowledged that there may come a point within the 20 week 
programme where pupils have completed all the levels on the 
decoding activities and that Teaching Assistants feel they have 
successfully completed the comprehension and reading activities. 
Assessment will be provided to ensure that Teaching Assistants feel 
confident when to discontinue pupils froom the programme. This 
assessment will be covered at the training event in October and further 
assistance can be given to the Teaching Assistants by a Senior 
Research Assistant if required. The discontinuation criteria for both the 
ICTand non-ICT interventions will be the same. 


Record Keeping 


Teaching Assistants willbe encouraged to keep a short record of every 
session they complete with pupils (see Appendix 7). This will be 
particulary important if they have to deviate from the lesson plans for 
any reason. 
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APPENDIX 1 - KEY FACTS SHEET 


Key Facts 


e The English language uses approximately 44 sounds (20 vowels 
and 24 consonants) 

e The English Alphabet has 26 letters 

e English uses combinations of letters to represent single sounds 

e There are 140 ways of combining letters to create the 44 sounds. 


leters:abcdefghlijkimnopqrstuvwxyz 


Phonemes: /b/ /k/ /d/ /ff/ /g/ /W/ /j/ /V /m/ /n/ /p/ / /S /¥ 
Iwi [we fy! (fa /shi [ch /th /th/ /ng/ /zh/ /a/ /e/ /i/ /o/ /uU/ ai/ 
/ee/ /igh/ /oa/ /o0/ /oo/ /ow/ /oi/ /ar/ /or/ /ur /air /ear /ure/ /er 


A phoneme is the smallest unit of sound in a word 


A grapheme is letter(s) representing a phoneme 


Phonemes are represented by letters (graphemes) 


e Achild need to leam the letters that make up each sounds, this is 
known as phoneme-grapheme representation 
e Phonemescan be in the initial, medial or final position of a word. 


A phoneme can be represented by one or more letters 
e Asingle phoneme represented by 2 lettersormore e.g. 


ch-ai-n 


A Diagraph is two letters which make one sound 
e A consonant diagraph contains two consonants 
sh ck th Il 


e A vowel diagraph contains at least one vowel also known as 
long vowel phoneme 


ai ee ar oy 


e Three letters, which make one sound 
igh dge 
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The same phoneme can be represented/ spelt in more than one way 
e This isparticulany common among the vowels 
rain, may, lake 


The same spelling may represent more than one phoneme 
e This is where children need to leam to use the skill of making 
sense of text 
mean, deaf 


Adjacent Consonants (formally known as blends) are _ letter 
combination where each letter makes an individual phoneme 


sp st sk so 
tr ft nt It 
mp un Ip dr 
cl sw cr sm 


Split Diagraphs are diagraphsin which the two letters are not adjacent 


ae ee ie oe ue 
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APPENDIX 2 


Blending and Decoding Tracking 
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Pupils work through the levels. Once they have achieved a correct response rate of 80-90% as a group for three consecutive entries you can 


move onto the next level. 


Week and Day 


Level 


Number 
Examples 


of Correct 


Number of Incorrect 
Examples 


Achieved (Y/N) 


Notes 
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APPENDIX 3 - BLENDING ACTIVITIES 


Teaching Assistants can chose from the range of activities depending 
on what they feel would be most suitable for their groups and the 
resources available in school. These different activities will be explored 
further in the training event. 


Magnetic letters 
e Facilitator places magnetic letters on a board with a gap 
between each phoneme. Support the pupils to sound out each 
phoneme and then push them together to create a word. If 
there are enough resources pupils could have their own board. 


Jigsaws 
e Facilitator to write letters onto jigsaw pieces. Support the pupils to 
sound out each letter. Then complete jigsaw and sound out full 
word. 


Train Camiages 
e Write each phoneme ona piece of card and attach to toy train 
camage. The facilitator should help the pupil to sound out each 
phoneme and blend them together to create a word. 


I Spy 
Play | Soy by sounding out the name of the object you are looking at. 
e Pupils have to blend the sounds together to determine the 
object 
e Eg.Ilspya/b/ /a/ /Il/ (ball). 
e If pupils require additional support you can add a clue to begin 
e.g. | soy something that is round. 


*This is only a sample of the activities which will be suggested to 
Teaching Assistant. Further examples willbe added. 
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APPENDIX 4 - WORD CHANGING EXAMPLES 
LEVEL 1 


fan/ man/can 
cat/ sat/ mat 
mad / dad / sad 
let/ met/ vet 


LEVEL 2 


big / bag / bad 
hit/ hat/ cat 

yet/ get/ got 
Dan/ dad / did 


LEVEL 3 


cage/ sage/ page 
bake / cake / rake 
mice / nice / rice 
dine / mine / fine 


LEVEL 4 
tide / time / tame 
dine / fine / find 


mine / pine / pile 
like / line / fine 
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ec Word Changing Tracking 


Week and Day | Level Number of Correct | Number of Incorrect | Achieved (Y/N) | Notes 
Examples Examples 


Education Endowment Foundation 94 


Pupils work through the levels. Once they have achieved a correct response rate of 80-90% as a group for three consecutive entries you can 
move onto the next level. 
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APPENDIX 6 - READING PROMPTS 


An example of a Reading Prompt sheet. These will be further developed. The 
Reading Prompt Sheets will be the same in both the Teaching Assistant led 
and computer package conditions. 


Top Tracking Tips 


e When accessing stories the facilitator first should read the required 
pages. 

e Feel free to ask questions about the illustrations, other prediction 
question, their opinion, their familiarity with the story. 

e The pupils will not be able to read many words and we do not want 
them to feel pressure to do so. 

e When finished reading the selected pages go back to the 
beginning. For each page ask if anyone can identify a word or two, 
or maybe even a letter if necessary. 

e Read each page. Ask the prediction / summary questions. 
Encourage each pupil to comment. 

e Use up any extra time by reviewing and discussing what has been 
read. 
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APPENDIX 7 - TEACHING ASSISTANT REC ORD SHEETS 


WEEK: PUPILS ATTENDED: 
DAY: 

ACTIVMES COMPLETED 
ACTIVITY NOTES 


ANY ISSUES? 
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Appendix C: Permission letters 


Coventry University NZ g 


Priory Street 
Coventry CV1 5FB 


Telephone 024 7688 7688 Covent 
Dr Linda Merriman Univers rv 


Dean of Faculty 


DATE. 
Dear Headteacher 


First of all, please allow me to apologise for contacting you somewhat out of the blue. My name is 
Prof. Clare Wood and | am the Director of the Centre for Applied Research in Psychology at Coventry 


University (www.coventry.ac.uk/CARP). 


Our team recently applied for funding from the Education Endowment Foundation for a project which 
would evaluate the educational impact of training Teaching Assistants to deliver Small Group-based 
Support for Literacy. We have been successful in securing funding for this project, and so we are now 
looking to talk to local schools who may be interested in participating in the project. The project itself 
would commence in September 2014, and the intervention phase would last for 20 weeks, starting 
after the Autumn half term break. 


| appreciate that you are likely to be very busy at this time of year, but if you think that you may be 
interested in discussing what would be involved from your point of view and how your school might 
benefit directly from participation, | would be very happy to meet with you or talk to you on the phone 
at a time which is convenient for you. | am very keen to involve as many Coventry schools as possible 
so that we have a strong local benefit and impact from the project, which is looking to recruit large 
numbers of children across the Midlands. 


If you think that you may be interested, | can be contacted via email on c.wood@coventry.ac.uk or by 
telephone (Monday — Thursday) on 02476 888226. Many thanks for time and | very much hope to 
hear from you in due course. 


Yours faithfully 


Prof. Clare Wood 
Professor of Psychology in Education 
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Coventry University SZ 


Priory Street 
Coventry CV1 5FB 


Telephone 024 7688 7688 Cove ntrv 
eee Univers ry 


Dean of Faculty 


12 June 2014 
Dear Parent 


My name is Dr Janet Vousden and | am a member of the Centre for Research in Psychology, 
Behaviour and Achievement (http://www.coventry.ac.uk/CRPBA). We recently secured funding from 
the Education Endowment Foundation for a large-scale project which will evaluate the educational 
impact of training Teaching Assistants to deliver small group support for reading in Year 1. This project 
will run across 60 schools in the West Midlands and your child’s school is participating in this project. | 
am therefore writing to you to make you aware of what is involved in the project and to offer you the 
chance to raise any questions about it with me directly, or with your school. Please feel free to contact 
me by email or phone if you have any concerns. | can be contacted via email 
on janet.vousden@coventry.ac.uk or by telephone (Monday — Thursday) on 02477 659510. 


| have attached an information sheet which explains in simple terms what is involved. We hope that as 
many pupils as possible will be able to participate but we also want to offer you the chance to opt out 
of the project, if you so wish. We will also be asking the individual children if they are happy to take 
part, and we will respect their wishes if the project is not for them. 


Please return the reply slip at the bottom of this letter to your school teacher as soon as possible and 
no later than the week before the end of term to notify me of your wishes. If we do not hear from you 


by this date we will assume that you have no objections and your child may be asked to take part. 


Yours faithfully 


Dr Janet Vousden 
Research Fellow in Reading Development 


Year 1 Reading Project 
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3] | confirm that | have read the participant information sheet sent by Dr Janet Vousden in 
relation to the Year 1 Reading project. 


[ | agree to my child being asked to participate in the project. 


L] | do not want my child to be asked to participate in this project. 


Child’s Name 


Parent's Name and Signature 
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Participant Information Sheet 


Study title: 
An Evaluation of Teaching Assistant-Based Small Group Support for Literacy 


What is the purpose of the study? 

We wish to evaluate the impact of two different approaches to supporting Year 1 children’s reading 
attainment through the provision of small group reading support delivered by trained Teaching 
Assistants. This project is being conducted across a large number of schools in the Midlands. 


Why have we been approached? 
We are looking to recruit Year 1 pupils from schools in the Midlands where the project team is based. 
This is the only reason why you have been approached. 


Do we have to take part? 
No, participation is entirely voluntary and there are no consequences if you or your child decides not to 
take part. 


What will happen to my child if s/he take part? 

If your child takes part, they will be assessed on his/her reading and language skills by standard tests 
provided by Hodder, an education publishing company. This will happen for all children at the 
beginning of Autumn Term (2014), at the end of Summer Term (2015) and again at the end of 
Summer Term (2016) to evaluate the impact of the study. Your school will be randomly allocated to 
either take part in the intervention in 2014/2015, or to be in a control group that will not be offered the 
intervention(s) until 2015/2016. If your child takes part, they will either: a) be randomly selected to 
experience either one of two different types of small group support for reading or normal classroom 
activities in 2014/2015, or b) experience either one of two different types of small group support for 
reading in 2015/2016. One small group support group will complete computer-based activities, the 
other will complete non-computer-based activities. The intervention will last for 20 weeks. 


What are the possible disadvantages and risks of taking part? 

There are few disadvantages and risks. The reading assessments do take a little time to complete but 
we will ensure that they are completed as a time when it will cause minimal disruption to your child’s 
school work. The assessments may feel challenging and some children may feel a little self-conscious 
about completing them (for example, if their reading is not as good as they would like it to be). 
However, all results will remain confidential to the research team and we will do our best to put your 
child at ease throughout. If your child is allocated to a control group, this may feel like a relative 
disadvantage, but it should be noted that there is no disadvantage insofar as the children in these 
groups will continue to receive the high quality literacy support that they would normally receive from 
their classroom teachers. 


What are the possible benefits of taking part? 

This project will enable schools to discover the best way of using Teaching Assistants to deliver more 
individualised teaching support for reading. We will provide opportunities for parents who are 
interested to find out more about the interventions that we are running and how they can also support 
their children’s work. We expect the results of this research to benefit your children’s school in future 
years, with consequences for how Teaching Assistants are subsequently used in the school to help 
support children’s reading. 


What if something goes wrong? 

You or your child can indicate to the teacher or the reading group leader if either of you no longer 
wishes to take part, and you can leave the study without question. If you are unhappy with the conduct 
of the study, you can contact me directly in the first instance using the number at the end of this sheet. 
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You are free to withdraw at any point during the study, and for up to one month following the 
completion of the study. You can do this by contacting me and giving me your child’s name and the 
name of his/ her school. If you are unhappy with the conduct of the research team, you can contact 
me (Dr Janet Vousden) in the first instance, or Prof. Neil Forbes, who is chair of the University Ethics 
Committee (address: Coventry University, Priory Street, Coventry, CV1 5FB). 


Will my taking part in this study be kept confidential? 

Yes. Pupils’ test responses and any other pupil data will be treated with the strictest confidence. The 
reading test responses will be scored by the service provided at Hodder and accessed by the team at 
London School of Economics. Named data will be matched with the National Pupil Database and 
shared with the Coventry University team, the independent evaluators at London School of Economics 
and the Education Endowment Foundation. We will not use your child’s name or the name of the 
school in any report arising from the research. Reading test sheets will be stored in a locked cabinet 
and destroyed as soon as the scores have been inputted into a computer file. All consent forms will be 
stored in a separate, locked location and will be destroyed on completion of the project. 


What will happen to the results of the research study? 

The reading test data will be used as the basis of a report to be compiled by colleagues from the 
London School of Economics, which will be submitted to the Education Endowment Foundation. 
Reports based on the reading scores will be presented at academic conferences and it may also be 
written up for publication in peer reviewed academic journals. Crucially, all participating schools and 
families will be informed as to the outcomes of the project overall, and we will host special events 
which families and non participating schools can attend to find out more about the project. 


Who is organising and funding the research? 
The research is organised by Dr Janet Vousden, who is a Research Fellow in the Centre for Applied 
Research in Psychology. It is funded by the Education Endowment Foundation. 


Who has reviewed the study? 
The Education Endowment Foundation and the Coventry University Ethics Committee have reviewed 
and approved this study. 


Contact for Further Information 
Dr Janet Vousden 

Psychology Department 
Coventry University 

Priory Street 

Coventry 

CV1 5FB. 


Tel: 024 77 659510 
Email: janet.vousden@coventry.ac.uk 
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School Location Size (pupils | Ofsted Number of | Number’ of | Number of | TA ICT session | Non-ICT 
in groups pupils in | TAs interviews observations | session 
intervention) intervention observations 

A Leicester 93 Requires 7 Non ICT 27 Control 2 ICT 2  Non-ICT | 2 2 

improvement 8 ICT 28:1CT 2Non-ICT | TA 
26 Non-ICT 2ICT TA 
B Coventry 17 Good 2 Non ICT 4 Control 1 ICT 1 = Non-ICT | 1 1 
Lict 6 ICT 1Non-ICT | TA 
5 Non-ICT LICT TA 
Cc Leicester 45 Requires 2 Non ICT 7 Control 1ICT 1 Non-ICT | 3 0 
improvement 3 ICT 9 ICT 1Non-ICT | TA 
8 Non-ICT 1 ICT 
TA/LSA 
D Birmingham | 120 Outstanding 10 Non ICT | 35 Control 2 ICT 1  Non-ICT | 3 1 
10 ICT 36 ICT 2Non-ICT | TA 
38 Non-ICT 2ICT TA 
E Coventry 60 Good 3 Non ICT 15 Control 2 ICT (job | 1 Non-ICT | 1 1 
3 ICT is ICT share) TA 
11 Non-ICT 1 Non-ICT 2ICT TA 
F Coventry 60 Good 3 Non ICT 8 Control 1 ICT 1ICT TA 3 0 
3 ICT 10 ICT 1 Non-ICT 1 = Non-ICT 
9 Non-ICT TA 
Total 395 55 17 16 13 5 
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Appendix E: Padlock rating 


Criteria for rating 


Attrition* 


0-10% 


Fair and clear quasi- 
experimental design for 


comparison (e.g. RDD) with Adjustment 
appropriate analysis, or MDES < for 
experimental design with 0.3 Balance 
minor concerns about 

validity [0] 


38 Well-matched comparison 
(using propensity score 
matching, or similar) or MDES < 
experimental design with 0.4 21-30% 
moderate concerns about 
validity : 
oe Adjustment |______ 
2@ | Weakly matched comparison for threats 
or experimental design with MDES < 31-40% to internal 
major flaws 0.5 validity 
eee | Ct C*~“‘;‘~sW‘C“C‘C(;‘SCNSCOCS:CO [oy j}— 
Comparison group with poor 
or no matching (E.g. MDES < 51-50% 
volunteer versus others) 0.6 
Oe 
No comparator we ? <50% 


e Initial padlock score: lowest of the three ratings for design, power and attrition = 4 padlocks 

e Reason for adjustment for balance (if made): none as the majority of pupil characteristics 
and all of the outcome measures are balanced at baseline. 

e Reason for adjustment for threats to validity (if made): none required 

e Final padlock score: initial score adjusted for balance and internal validity = 4 padlocks 


*Attrition should be measured at the pupil level, even for cluster trials. 
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Appendix F: Cost rating 


Cost ratings are based on the approximate cost per pupil per year of implementing the intervention 
over three years. More information about the EEF’s approach to cost evaluation can be found here. 
Cost ratings are awarded as follows: 


Cost rating Description 
Very low: less than £80 per pupil per year. 
Low: up to about £200 per pupil per year. 


Moderate: up to about £700 per pupil per year. 
High: up to £1,200 per pupil per year. 
Very high: over £1,200 per pupil per year. 
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You may re-use this document/publication (not including logos) free of charge in any format or 
medium, under the terms of the Open Government Licence v2.0. 


To view this licence, visit www .nationalarchives. gov.uk/doc/open-government-icence/version/2 or 


email: psi@nationalarchives. gsi.gov.uk 


Where we have identified any third party copyright information you will need to obtain permission 
from the copyright holders concerned. The views expressed in this report are the authors’ and do 
not necessarily reflect those of the Department for Education. 


This document is available for download at www.educationendowmentfoundation.org.uk 
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